Posts tagged RTTI

Variant Invoke and RTTI

I was “spring cleaning” my temporary project directory on my hard drive today and I found a half-finished implementation of a dynamic dispatch for objects based on new RTTI (Delphi 2010+). I spent some time and fixed the last issues I had and now I think it’s ready to be published.

All the code is enclosed in a single unit (there’s not that much code actually) that exposes one function called “AsDynamic“. This method expects a normal non-nil object, a lifetime management flag and returns a custom Variant value.  The Variant can be used later on to dynamically invoke any method/property on that object.

The following example shows a wrapped TStringList object:

var
  LVarList: Variant;
begin
  { Create a new String List and wrap it into a Variant. By default the Variant will also take care about freeing the object. }
  LVarList := AsDynamic(TStringList.Create);

  { Add two strings to the string list. Note that Add() is called dynamically here. }
  LVarList.Add('Hello World!');
  LVarList.Add('Cool string');

  { Make the list sorted (also dynamically) }
  LVarList.Sorted := True;

  { ... And finally write the contents of the list }
  WriteLn(LVarList.Text);
end.

I don’t see any practical uses this code but it’s good for learning purposes. It can also be a simplified way of treating the cases when one has to execute the same code on classes that do not share a common ancestor or cannot implement an interface.

Just a quick note on what you can see inside the unit:

  • Defining a custom Variant type by extending TInvokeableVariantType and providing the proper support code.
  • An intermediate object and an interface that contain the wrapped object and RTTI structures needed for dynamic invoke. The custom Variant type carries over this object (actually an interface reference).
  • TValue to Variant and Variant to TValue conversion. Since the dynamic dispatch uses variants for arguments a conversion is required in both directions. This is simple in most cases but rather complicated for var and out parameters.

Note that dynamic dispatch as implemented by Delphi has quite a few restriction in what types are allowed and how exactly those values are passed to the invoked methods. For example literals are passed by value and variables are passed by reference (not always!). This is due to the nature of “not knowing” what exactly the method behind expects. As an example, you cannot pass a String by reference while a WideString variable is always passed by reference.

The code can be found here: Dynamic Object Dispatch (2301)

TZDB 1.7.110207 and Collections 1.1.1

TZDB 1.7.110207

No code changes, just an updated DB from “The tz Database“. There are a small number of changes there so upgrade only if you want to stay 100% up to date. I would just recommend you skip it. As usual, you can get the latest version here.

Collections 1.1.1

This is a bug fix release that addresses the following issues:

  1. No possibility to remove an value from a list or associative collection without invoking the “removal procedures”. This results in freed objects for object based collections. In pre-1.1 versions this wasn’t necessarily an issue. (Thanks Mason for the report, hope this unblocks your scripting project)
  2. Removal procedures were not invoked for bidirectional dictionaries.
  3. TObjectMultiMap<TKey, TValue> still used a “class” restriction on TValue type parameter.
  4. Fixed Delphi 2010 build. Note that by “fix” I mean — disabled Collections.Dynamic code and TBitSet class. Cannot make the compiler accept those (circular unit problems that were fixed in XE).

Get the bug fix release here.

DeHL 0.8.3

DeHL

I will be brief as usual — version 0.8.3 of DeHL is out. The downloads can be found on this page and changelog on this page. This release “fixes” some of the things I wanted fixed for a long time, so it seemed this is the perfect moment for this to happen. A new unit is introduced — DeHL.Tuples — which brings seven generic Tuple<…> types. I have also finished moving away from Integer and Cardinal to NativeInt and NativeUInt through all DeHL.

Breaking changes are:

  1. DeHL.Converter was renamed to DeHL.Conversion. This name change was done mostly to reflect the nature of the unit. It does not contain one simple class anymore. Now there is a fully featured conversion system.
  2. TBinaryTree<T> (in DeHL.Collections.BinaryTree) is no more. It was written way in the beginnings of the project and was buggy, incomplete and utterly useless.
  3. And the most visible and breaking of all changes is the removal of TKeyValuePair<TKey, TValue> (in DeHL.KeyValuePair). It was replaced with KVPair<TKey, TValue> (in DeHL.Tuples). The easiet way to get over this change is to find and replace all TKeyValuePair instances with KVPair and all DeHL.KeyValuePair uses with DeHL.Tuples.

Now, obviously an example using Tuples:

uses
  SysUtils, DeHL.Tuples;

{ If you are lazy and do not wish to declare a new record type
  to be used as result. Use Tuple<..> do to that. }
function GiveMeSomeData(): Tuple<String, Integer, Integer>;
begin
  { ... Do some processing ... }
  Result := Tuple.Create('Some data', 100, -99);
end;

var
  LResult: Tuple<String, Integer, Integer>;
begin
  { Obtain the result }
  LResult := GiveMeSomeData();

  { And write the results to the console }
  with LResult do
    WriteLn('Something was done with result: ', Value1, ', ', 
      Value2, ', ', Value3, '!');
end.

The new conversion engine handles most of the possible conversions and also allows registering custom ones:

uses
  SysUtils,
  DeHL.Conversion,
  DeHL.Collections.List;

type
  { Declare some type that cannot be converted into integer directly.
    We're making up an "int". }
  TRecordInt = packed record
    FValueAsString: string;
    FSign: Boolean;

    constructor Create(const AInt: Integer);
  end;

{ TRecordInt }

constructor TRecordInt.Create(const AInt: Integer);
begin
  { Decompose an int }
  FValueAsString := IntToStr(Abs(AInt));
  FSign := (AInt < 0);
end;

var
  LInputList: TList<TRecordInt>;
  S: String;
  I: Integer;
begin
  { Create a list of TRecordInt }
  LInputList := TList<TRecordInt>.Create();

  { Fill it with some random values (positive and negative) }
  for I := 0 to 10 do
    LInputList.Add(TRecordInt.Create(Random(MaxInt) - (MaxInt div 2)));

  { Now comes the interesting part ... register a custom converter
    from TRecordInt to Integer }
  TConverter<TRecordInt, Integer>.Method :=
    function(const AIn: TRecordInt; out AOut: Integer): Boolean
    begin
      { Convert the TRecordInt back to an integer }
      Result := TryStrToInt(AIn.FValueAsString, AOut);
      if Result and AIn.FSign then
        AOut := -AOut;
    end;

  { Now print the values to the console. Convert them from TRecordInt to
    Integer then to String }
  for S in LInputList.Op.Cast<Integer>.Op.Cast<String> do
    WriteLn(S);
end.

TConverter is also smart enough to figure out that “type MyInt = type Integer” is actually equivalent to Integer. If there is no explicit custom conversion method registered for it the converter for the standard type will be selected is possible. In the worst case, when TConverter cannot convert directly between the given types, it falls back to Variant conversion (using TType.TryConvertToVariant and TType.TryConvertFromVariant) which all types registered with DeHL’s type system, if possible, should implement.

Well, that’s all for today,
Have Fun!

DeHL 0.8.2 is out

DeHL

I’ve just released the version 0.8.2 of DeHL. The downloads can be found on this page and changelog on this page.
Again, this is a minor release with a few bugs fixed and a new feature: TString (as asked in this comment).
As you might have guessed already, TString is a wrapper record modeled on .NET’s System.String class. Unfortunately I was unable to use most of the RTL’s string functionality so the “wrapper” grew quite a bit from my original expectations.

But … enough talk, here are some usage scenarios that someone may find useful:

var
  LStr: TString;
begin
  { Overloaded operators and a special function "U" }
  LStr := U('Hello World for the ') + 10 + 'th time!';

  { Do some random operations }
  if (LStr.ToUpper().Contains('HELLO')) and
     (LStr.Contains('HeLLo', scLocaleIgnoreCase)) then
  WriteLn(LStr.ToString);

  { Now let's select all the distinct chars from the string }
  WriteLn(
    LStr.Concat(LStr.AsCollection.Distinct.Op.Cast<string>).ToString
  );
end.

TString overloads all sane operators: Equality, Inequality, Implicit conversions, Addition, Subtraction and offers functions to convert to and from UTF8 and UCS4 (via RTL of course). I also need to iron a few things about about Enex integration for the next minor release.

The other small improvement that I added relates to the collection package. All simple collections (not the Key/Value pair ones) implement a sort of “where T is the_class, select it as such” operation. Check out this example:

var
  LList: TList<TObject>;
  LBuilder: TStringBuilder;
  LObject: TObject;
begin
  LList := TList<TObject>.Create;

  { Populate the list with some random objects }
  LList.Add(TInterfacedObject.Create);
  LList.Add(TStringBuilder.Create);
  LList.Add(TStringBuilder.Create);
  LList.Add(TObject.Create);

  { Now select the objects we're interested in (string builders) }
  for LBuilder in LList.Op.Select<TStringBuilder> do
    WriteLn(LBuilder.ClassName); // Do stuff

  { Or select everything (not actually required - an example) }
  for LObject in LList.Op.Select<TObject> do
    WriteLn(LObject.ClassName); // Do stuff
end.

If it’s still not clear what this operations does, let me explain. It basically consists of two operations: Where and Select. First, each object is checked to be of a given class and then this object is cast to that class so you can iterate directly using a FOR .. IN loop only over the objects you want to. Of course doing that for TObject makes no sense (as in example) … but well … that was an example.

Well, that’s all for today,
Have Fun!

DeHL 0.8.1

DeHL

I’ve just released the version 0.8.1 of DeHL. The downloads can be found on this page and changelog on this page.

This is mostly a fix release with only one major feature – Cloning (in DeHL.Cloning). The rest of the changes are either bug fixes or janitorial changes.

Have Fun!

Replication

I must say, I am still pretty exited by the extended RTTI in Delphi 2010. It makes life so much easier in many circumstances. “Cloning” (I call it replication) is one of those. Say hello to TReplicator<T> (in DeHL.Replication). It can take any type and create an exact copy (including deep cloning for complex types).

The following example should pretty much explain what and how:

uses
  SysUtils,
  DeHL.Replication,
  DeHL.Collections.List;

type
  { Define a test class }
  TData = class
    { Pointer to self }
    FSelf: TObject;

    { A chunck of bytes }
    FSomeData: TBytes;

    { Some dummy string }
    FName: string;

    { An internal pointer. Marked as non-replicable -- will not be copied }
    [NonReplicable]
    FInternalPtr: Pointer;

    { This field will be copied by reference. It means that the new
      object will have the reference to the same TList<String> and not
      a copy of it. }
    [ByReference]
    FList: TList<String>;
  end;

var
  LInput, LOutput: TData;
  LReplicator: TReplicator<TData>;
begin
  { Create the input }
  LInput := TData.Create;
  LInput.FSelf := LInput;
  LInput.FSomeData := TBytes.Create(1, 2, 3, 4, 5, 6);
  LInput.FName := 'Some Name';
  LInput.FInternalPtr := Ptr($DEADBABE);
  LInput.FList := TList<String>.Create(['Hello', 'World']);

  { Create the replicator }
  LReplicator := TReplicator<TData>.Create;

  { Create a copy of LInput }
  LReplicator.Replicate(LInput, LOutput);

  { ... }
end.

As you can see we take LInput and create a copy into LOutput. Also note the use of [NonReplicable] and [ByReference] attributes on two fields of TData class. NonReplicable forces the replication process to skip the field and ByReference forces a shallow copy of reference fields (objects, pointers to arrays and dynamic arrays).

For the ease of use, a TCloneableObject is defined in the same unit (DeHL.Replication). TCloneableObject provides a method called Clone() which returns a copy of the object:

uses
  SysUtils,
  DeHL.Replication;

type
  { Define a test class }
  TTestMe = class(TCloneableObject)
    { ... }
  end;

var
  LInput, LOutput: TTestMe;
begin
  { ... }
  LOutput := LInput.Clone() as TTestMe;
  { ... }
end.

Note: This is just a preview of a new feature finding it’s way into DeHL. It’s currently only in the SVN repository. As usual, I need to write unit tests and so on to make sure it actually has no bugs (or at leas not that many bugs ;).

Here you go — DeHL 0.8

DeHL

This is going to be a short one. After months of no releases, here it is: DeHL 0.8 (see changelog for the list of changes on this release). As I mentioned previously, this release will only work on Delphi 2010, since the number of changes required to support serialization was quite big.

I must confess, this is the first time in my programming career that I had to interact with serialization (excluding the times when the frameworks do that for you). I had to learn quite a bit and changed my internal design three times.

So, here’s an example how to serialize a TFormatSettings structure into binary format:

var
  LBinFile: TFileStream;
  LBinSerializer: TBinarySerializer<TFormatSettings>;
  LSettings: TFormatSettings;
begin
  { Obtain current thread's format settings }
  GetFormatSettings(GetThreadLocale(), LSettings);

  { Create a serializer and a file stream }
  LBinSerializer := TBinarySerializer<TFormatSettings>.Create();
  LBinFile := TFileStream.Create('dump_obj.bin', fmCreate);

  { Serialize the structure }
  try
    LBinSerializer.Serialize(LSettings, LBinFile);
  finally
    LBinFile.Free;
    LBinSerializer.Free;
  end;
end;

… and here’s an example how to deserialize it:

var
  LBinFile: TFileStream;
  LBinSerializer: TBinarySerializer<TFormatSettings>;
  LSettings: TFormatSettings;
begin
  { Create a serializer and a file stream }
  LBinSerializer := TBinarySerializer<TFormatSettings>.Create();
  LBinFile := TFileStream.Create('dump_obj.bin', fmOpenRead);

  { Deserialize the structure }
  try
    LBinSerializer.Deserialize(LSettings, LBinFile);
  finally
    LBinFile.Free;
    LBinSerializer.Free;
  end;
end;

Note: I have written quite a few unit tests to support the new changes, but most certainly there are hidden bugs. If you find one please report it here.

Other Note: I exhausted my idea jar regarding new features. If you have an idea please do not hesitate to drop me a comment or an email.

Have Fun!

DeHL, Delphi 2010 and Serialization

DeHLA few months have passed and I did not release a new version of DeHL yet. No, it’s not dead. I’ve just been busy with a delicate new feature — Serialization. This post will demonstrate the new capabilities of DeHL it’s advantages and and shortcomings.

But first — since the new releases will focus mostly on serialization and related stuff, I decided to drop Delphi 2009 support. It made no sense to support 2009 for future versions since no essential changes are made to the prior code. You can still use 0.7 release in Delphi 2009.

Back to serialization. The following list describes the changes that went into the new version:

  • In order to support serialization, DeHL’s type system was extended to support Serialize and Deserialize methods. Each type class (that describes a type in Delphi) now knows how to serialize values of the type it manages.
  • A new unit named DeHL.Serialization was added. It contains the base definitions of types used by the type system for serialization.
  • TPointerType, TRecordType<T>, TArrayType<T>, etc. were added for simplified type handling. The old method was a mix-up of Delphi 2009 and Delphi 2010 RTTI specifics (which have some essential differences in my case).
  • All classes can now implement ISerializable interface. The TClassType<T> detects whether this interface is implemented by the object and uses it for serialization (no, reference counting is not touched).
  • DeHL.Serialization.Abstract contains the semi-implementation of a “serializer” and it’s context. It is used by specific serializers.
  • DeHL.Serialization.XML defines the TXMLSerializer<T> which can be used to serialize/deserialize into XML nodes (uses TXMLDocument). Supports it’s own set of attributes (such as XmlRoot, XmlElement, etc.).
  • DeHL.Serialization.Ini defines the TIniSerializer<T> that you can use to serialize/deserialize type into Ini files or registry (through RTL’s TRegIniFile).
  • Most DeHL types (such as Nullable<T>TFixedArray<T>, BigInteger, etc.) provide their own serialization and deserialization methods.
  • All Enex collections (except a few that can’t actually) can be serialized and deserialized. They implement a custom serialization and deserialization technique through ISerializable.

Enough talk, a mandatory example:

type
  [XmlRoot('Testing', 'http://test.namespace.com')]
  TTest = class
    { Pointer to self }
    [XmlElement('PointerToSelf')]
    FSelf: TObject; 
    {A set of format settings }
    FFormatSettings: TFormatSettings;

    { And internal record }
    FInternal: record
      { Force the field to be an attribute of FInternal }
      [XmlAttribute('Value')]
      FOne: Integer;

      { Force this element to have same name but other namespace }
      [XmlElement('Value', 'http://other.namespace.com')]
      FTwo: String;
    end;

    FListOfDoubles: TList<Double>;
  end;
var
  LDocument: IXMLDocument;
  LXMLSerializer: TXMLSerializer<TTest>;
  LOutInst, LInInst: TTest;
begin
  CoInitializeEx(nil, 0);

  { Initialize the test object }
  LOutInst := TTest.Create;
  LOutInst.FSelf := LOutInst;
  GetLocaleFormatSettings(GetThreadLocale(), LOutInst.FFormatSettings);
  LOutInst.FInternal.FOne := 1;
  LOutInst.FInternal.FTwo := '2 - Two';
  LOutInst.FListOfDoubles := TList<Double>.Create();
  LOutInst.FListOfDoubles.Add(0.55);
  LOutInst.FListOfDoubles.Add(0.122);
  LOutInst.FListOfDoubles.Add(122.23);

  { Create the serializer and an XML document }
  LXMLSerializer := TXMLSerializer<TTest>.Create();
  LDocument := TXMLDocument.Create(nil);

  { Set the options }
  LDocument.Active := true;
  LDocument.Options := LDocument.Options + [doNodeAutoIndent];

  { Force fields to elements by default }
  LXMLSerializer.DefaultFieldsToTags := true;

  { Serialize the structure }
  LXMLSerializer.Serialize(LOutInst, LDocument.Node);

  { Serialize the structure }
  LXMLSerializer.Deserialize(LInInst, LDocument.Node);

  { Cleanup }
  LDocument.SaveToFile('c:\test.xml');
  LXMLSerializer.Free;
end.

The XML file generated by this code looks like this (INI looks uglier):

<Testing xmlns="http://test.namespace.com" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:DeHL="http://alex.ciobanu.org/DeHL.Serialization.XML" xmlns:NS1="http://other.namespace.com">
  <PointerToSelf DeHL:ref="Testing"/>
  <FFormatSettings>
    <CurrencyString>$</CurrencyString>
    <CurrencyFormat>0</CurrencyFormat>
    <CurrencyDecimals>2</CurrencyDecimals>
    <DateSeparator>/</DateSeparator>
    <TimeSeparator>:</TimeSeparator>
    <ListSeparator>,</ListSeparator>
    <ShortDateFormat>M/d/yyyy</ShortDateFormat>
    <LongDateFormat>dddd, MMMM dd, yyyy</LongDateFormat>
    <TimeAMString>AM</TimeAMString>
    <TimePMString>PM</TimePMString>
    <ShortTimeFormat>h:mm AMPM</ShortTimeFormat>
    <LongTimeFormat>h:mm:ss AMPM</LongTimeFormat>
    <ShortMonthNames>
      <string>Jan</string>
      <string>Feb</string>
      <string>Mar</string>
      <string>Apr</string>
      <string>May</string>
      <string>Jun</string>
      <string>Jul</string>
      <string>Aug</string>
      <string>Sep</string>
      <string>Oct</string>
      <string>Nov</string>
      <string>Dec</string>
    </ShortMonthNames>
    <LongMonthNames>
      <string>January</string>
      <string>February</string>
      <string>March</string>
      <string>April</string>
      <string>May</string>
      <string>June</string>
      <string>July</string>
      <string>August</string>
      <string>September</string>
      <string>October</string>
      <string>November</string>
      <string>December</string>
    </LongMonthNames>
    <ShortDayNames>
      <string>Sun</string>
      <string>Mon</string>
      <string>Tue</string>
      <string>Wed</string>
      <string>Thu</string>
      <string>Fri</string>
      <string>Sat</string>
    </ShortDayNames>
    <LongDayNames>
      <string>Sunday</string>
      <string>Monday</string>
      <string>Tuesday</string>
      <string>Wednesday</string>
      <string>Thursday</string>
      <string>Friday</string>
      <string>Saturday</string>
    </LongDayNames>
    <ThousandSeparator>,</ThousandSeparator>
    <DecimalSeparator>.</DecimalSeparator>
    <TwoDigitYearCenturyWindow>50</TwoDigitYearCenturyWindow>
    <NegCurrFormat>0</NegCurrFormat>
  </FFormatSettings>
  <FInternal Value="1">
    <NS1:Value>2 - Two</NS1:Value>
  </FInternal>
  <FListOfDoubles>
    <Elements>
      <Double>0.55</Double>
      <Double>0.122</Double>
      <Double>122.23</Double>
    </Elements>
  </FListOfDoubles>
</Testing>

On the first serialized/deserialized value, serializers build up a sort of an internal “object graph” and gathers all information about the data being serialized. The next uses of the same serializer instance yield an 10x performance gain since there is no need to rebuild all the information from scratch. I am still working on more optimizations that could give greater speed boost.

P.S. I can’t show the contents of the deserialized object here so you’ll have to take my word for it.

Note. This is just a preview of what is going on in the trunk. No version is released since I have to iron out the last problems and write the missing unit tests.



TypeInfo workaround

This is going to be a short one. Just wanted to share a simple and elegant work-around for this QC issue:

type
  TypeOf<T> = record
    class function TypeInfo: PTypeInfo; static;
    class function Name: string; static;
    class function Kind: TTypeKind; static;
  end;

{ TypeOf<T> }

class function TypeOf<T>.Kind: TTypeKind;
var
  LTypeInfo: PTypeInfo;
begin
  LTypeInfo := TypeInfo;

  if LTypeInfo <> nil then
    Result := LTypeInfo^.Kind
  else
    Result := tkUnknown;
end;

class function TypeOf<T>.Name: string;
var
  LTypeInfo: PTypeInfo;
begin
  LTypeInfo := TypeInfo;

  if LTypeInfo <> nil then
    Result := GetTypeName(LTypeInfo)
  else
    Result := '';
end;

class function TypeOf<T>.TypeInfo: PTypeInfo;
begin
  Result := System.TypeInfo(T);
end;

Now, you can obtain the type information for any type by simply using TypeOf<T>.TypeInfo:

type
  TRecord<T> = record
    FVal: T;
  end;

  TIntRecord = TRecord<Integer>;

begin
  // TypeInfo(TIntRecord); // Fails with compile-time error;
  WriteLn(GetTypeName(TypeOf<TIntRecord>.TypeInfo));

  ReadLn;
end.

You can also use this construct to obtain the type information for generic types from within themselves:

type
  TSomeRec<T> = record
    function GetMyName: String;
  end;

function TSomeRec<T>.GetMyName(): String;
begin
  // Result := GetTypeName(TypeInfo(TSomeRec<T>)); // Fails
  Result := TypeOf<TSomeRec<T>>.Name;
end;

The only thing to notice is that compile time [DCC Error] E2134 Type ‘XXX’ has no type info errors will not be triggered for types having no type info. Instead, the TypeOf<T>.TypeInfo method will return nil.

DeHL 0.6 available

DeHL

Yes I know I have skipped 0.5. The reality is that 0.5 was due a long time ago, but I did not have enough free time on my hands to complete the unit testing for all new features added. I always try to add as many unit tests as possible to test all possible scenarios.

This release features a general reorganization of some key concepts, new type extension mechanics, boxing, a lot more collections, better support for associative collections through specific Enex extensions … and many more. I am not going to lay out all the changes here since there are quite a few of them. You can view the change log here.

One important note, check out the DeHL.Base.pas unit. There are a number of TOGGLE_XXXXX constants. Toggle them to true or false (if you are using Weaver FT builds) to enable or disable specific functionality. In Tiburon (Delphi 2009) all the features are disabled by default (to be on the safe side).

Get the latest build here. If you stumble upon some bugs please report them here.

Have Fun!