YAPB

A few months have passed and I did not release a new version of DeHL yet. No, it’s not dead. I’ve just been busy with a delicate new feature — Serialization. This post will demonstrate the new capabilities of DeHL it’s advantages and and shortcomings.

But first — since the new releases will focus mostly on serialization and related stuff, I decided to drop Delphi 2009 support. It made no sense to support 2009 for future versions since no essential changes are made to the prior code. You can still use 0.7 release in Delphi 2009.

Back to serialization. The following list describes the changes that went into the new version:

• In order to support serialization, DeHL’s type system was extended to support Serialize and Deserialize methods. Each type class (that describes a type in Delphi) now knows how to serialize values of the type it manages.
• A new unit named DeHL.Serialization was added. It contains the base definitions of types used by the type system for serialization.
• TPointerType, TRecordType<T>, TArrayType<T>, etc. were added for simplified type handling. The old method was a mix-up of Delphi 2009 and Delphi 2010 RTTI specifics (which have some essential differences in my case).
• All classes can now implement ISerializable interface. The TClassType<T> detects whether this interface is implemented by the object and uses it for serialization (no, reference counting is not touched).
• DeHL.Serialization.Abstract contains the semi-implementation of a “serializer” and it’s context. It is used by specific serializers.
• DeHL.Serialization.XML defines the TXMLSerializer<T> which can be used to serialize/deserialize into XML nodes (uses TXMLDocument). Supports it’s own set of attributes (such as XmlRoot, XmlElement, etc.).
• DeHL.Serialization.Ini defines the TIniSerializer<T> that you can use to serialize/deserialize type into Ini files or registry (through RTL’s TRegIniFile).
• Most DeHL types (such as Nullable<T>,  TFixedArray<T>, BigInteger, etc.) provide their own serialization and deserialization methods.
• All Enex collections (except a few that can’t actually) can be serialized and deserialized. They implement a custom serialization and deserialization technique through ISerializable.

Enough talk, a mandatory example:

type
  [XmlRoot('Testing', 'http://test.namespace.com')]
  TTest = class
    { Pointer to self }
    [XmlElement('PointerToSelf')]
    FSelf: TObject;
    {A set of format settings }
    FFormatSettings: TFormatSettings;

    { And internal record }
    FInternal: record
      { Force the field to be an attribute of FInternal }
      [XmlAttribute('Value')]
      FOne: Integer;

      { Force this element to have same name but other namespace }
      [XmlElement('Value', 'http://other.namespace.com')]
      FTwo: String;
    end;

    FListOfDoubles: TList<Double>;
  end;
var
  LDocument: IXMLDocument;
  LXMLSerializer: TXMLSerializer<TTest>;
  LOutInst, LInInst: TTest;
begin
  CoInitializeEx(nil, 0);

  { Initialize the test object }
  LOutInst := TTest.Create;
  LOutInst.FSelf := LOutInst;
  GetLocaleFormatSettings(GetThreadLocale(), LOutInst.FFormatSettings);
  LOutInst.FInternal.FOne := 1;
  LOutInst.FInternal.FTwo := '2 - Two';
  LOutInst.FListOfDoubles := TList<Double>.Create();
  LOutInst.FListOfDoubles.Add(0.55);
  LOutInst.FListOfDoubles.Add(0.122);
  LOutInst.FListOfDoubles.Add(122.23);

  { Create the serializer and an XML document }
  LXMLSerializer := TXMLSerializer<TTest>.Create();
  LDocument := TXMLDocument.Create(nil);

  { Set the options }
  LDocument.Active := true;
  LDocument.Options := LDocument.Options + [doNodeAutoIndent];

  { Force fields to elements by default }
  LXMLSerializer.DefaultFieldsToTags := true;

  { Serialize the structure }
  LXMLSerializer.Serialize(LOutInst, LDocument.Node);

  { Serialize the structure }
  LXMLSerializer.Deserialize(LInInst, LDocument.Node);

  { Cleanup }
  LDocument.SaveToFile('c:\test.xml');
  LXMLSerializer.Free;
end.

The XML file generated by this code looks like this (INI looks uglier):

<Testing xmlns="http://test.namespace.com" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:DeHL="http://alex.ciobanu.org/DeHL.Serialization.XML" xmlns:NS1="http://other.namespace.com">
  <PointerToSelf DeHL:ref="Testing"/>
  <FFormatSettings>
    <CurrencyString>$</CurrencyString>
    <CurrencyFormat>0</CurrencyFormat>
    <CurrencyDecimals>2</CurrencyDecimals>
    <DateSeparator>/</DateSeparator>
    <TimeSeparator>:</TimeSeparator>
    <ListSeparator>,</ListSeparator>
    <ShortDateFormat>M/d/yyyy</ShortDateFormat>
    <LongDateFormat>dddd, MMMM dd, yyyy</LongDateFormat>
    <TimeAMString>AM</TimeAMString>
    <TimePMString>PM</TimePMString>
    <ShortTimeFormat>h:mm AMPM</ShortTimeFormat>
    <LongTimeFormat>h:mm:ss AMPM</LongTimeFormat>
    <ShortMonthNames>
      <string>Jan</string>
      <string>Feb</string>
      <string>Mar</string>
      <string>Apr</string>
      <string>May</string>
      <string>Jun</string>
      <string>Jul</string>
      <string>Aug</string>
      <string>Sep</string>
      <string>Oct</string>
      <string>Nov</string>
      <string>Dec</string>
    </ShortMonthNames>
    <LongMonthNames>
      <string>January</string>
      <string>February</string>
      <string>March</string>
      <string>April</string>
      <string>May</string>
      <string>June</string>
      <string>July</string>
      <string>August</string>
      <string>September</string>
      <string>October</string>
      <string>November</string>
      <string>December</string>
    </LongMonthNames>
    <ShortDayNames>
      <string>Sun</string>
      <string>Mon</string>
      <string>Tue</string>
      <string>Wed</string>
      <string>Thu</string>
      <string>Fri</string>
      <string>Sat</string>
    </ShortDayNames>
    <LongDayNames>
      <string>Sunday</string>
      <string>Monday</string>
      <string>Tuesday</string>
      <string>Wednesday</string>
      <string>Thursday</string>
      <string>Friday</string>
      <string>Saturday</string>
    </LongDayNames>
    <ThousandSeparator>,</ThousandSeparator>
    <DecimalSeparator>.</DecimalSeparator>
    <TwoDigitYearCenturyWindow>50</TwoDigitYearCenturyWindow>
    <NegCurrFormat>0</NegCurrFormat>
  </FFormatSettings>
  <FInternal Value="1">
    <NS1:Value>2 - Two</NS1:Value>
  </FInternal>
  <FListOfDoubles>
    <Elements>
      <Double>0.55</Double>
      <Double>0.122</Double>
      <Double>122.23</Double>
    </Elements>
  </FListOfDoubles>
</Testing>

On the first serialized/deserialized value, serializers build up a sort of an internal “object graph” and gathers all information about the data being serialized. The next uses of the same serializer instance yield an 10x performance gain since there is no need to rebuild all the information from scratch. I am still working on more optimizations that could give greater speed boost.

P.S. I can’t show the contents of the deserialized object here so you’ll have to take my word for it.

Note. This is just a preview of what is going on in the trunk. No version is released since I have to iron out the last problems and write the missing unit tests.



This is going to be a short one. Just wanted to share a simple and elegant work-around for this QC issue:

type
  TypeOf<T> = record
    class function TypeInfo: PTypeInfo; static;
    class function Name: string; static;
    class function Kind: TTypeKind; static;
  end;

{ TypeOf<T> }

class function TypeOf<T>.Kind: TTypeKind;
var
  LTypeInfo: PTypeInfo;
begin
  LTypeInfo := TypeInfo;

  if LTypeInfo <> nil then
    Result := LTypeInfo^.Kind
  else
    Result := tkUnknown;
end;

class function TypeOf<T>.Name: string;
var
  LTypeInfo: PTypeInfo;
begin
  LTypeInfo := TypeInfo;

  if LTypeInfo <> nil then
    Result := GetTypeName(LTypeInfo)
  else
    Result := '';
end;

class function TypeOf<T>.TypeInfo: PTypeInfo;
begin
  Result := System.TypeInfo(T);
end;

Now, you can obtain the type information for any type by simply using TypeOf<T>.TypeInfo:

type
  TRecord<T> = record
    FVal: T;
  end;

  TIntRecord = TRecord<Integer>;

begin
  // TypeInfo(TIntRecord); // Fails with compile-time error;
  WriteLn(GetTypeName(TypeOf<TIntRecord>.TypeInfo));

  ReadLn;
end.

You can also use this construct to obtain the type information for generic types from within themselves:

type
  TSomeRec<T> = record
    function GetMyName: String;
  end;

function TSomeRec<T>.GetMyName(): String;
begin
  // Result := GetTypeName(TypeInfo(TSomeRec<T>)); // Fails
  Result := TypeOf<TSomeRec<T>>.Name;
end;

The only thing to notice is that compile time [DCC Error] E2134 Type ‘XXX’ has no type info errors will not be triggered for types having no type info. Instead, the TypeOf<T>.TypeInfo method will return nil.

After a few months of no releases, I finally decided to throw one out — so here it is, DeHL 0.7. This release is adding three more collection, new types and fixes some internal limitations of the library.

For the people that never tried DeHL – it is a collection of types and classes that use (and even abuse) the newer generation Delphi compilers into obtaining some features that were impossible in the past releases.

The newly added collection classes are:

• TPriorityQueue<TPriority, TValue> which implements the priority queue.
• TDistinctMultiMap<TKey, TValue> that is similar to TMultiMap but uses sets instead of lists to store the values. This ensures that all values for a key are distinct. As usual 2 more flavors of this class exist – TSortedDistinctMultiMap<TKey, TValue> and TDoubleSortedDistinctMultiMap<TKey, TValue>.
• TBidiMap<TKey, TValue> implements the concept of bi-directional map in which there is a bi-directional relation between keys and values (both are actually keys). TSortedBidiMap<TKey, TValue> and TDoubleSortedBidiMap<TKey, TValue> are the other two flavors of this class.
• TStringList<T> and TWideStringList<T> collections are actually generic variants of the normal TStringList and TWideStringList (which are used as base classes).

Other enhancements include:

• Singleton<T> class. Which can be used to access the same instance of a class across all application.
• TWideCharSet (this was inspired on other implementations out there but use dynamic arrays to save memory and speed in certain circumstances).
• TType speed and reliability increases.
• A global DeHL.Defines.inc files for all $IFDEF needs.
• Types such as Nullable<T>, Scoped<T> and TKeyValuePair<TKey, TValue> now have their own type classes.

For a full list of changes see the changelog.

This release also makes use of some new improvements of the Delphi 2010 compiler (such as class constructors, or some fixed problems in the generics handling) which makes some features unavailable on Delphi 2009. I plan to keep Delphi 2009 supported for as long as possible, but many new planned features are not going to be available on that platform.

Have Fun!

The new great addition to the Delphi language in Delphi 2010 is the possibility to specify a class constructor and a class destructor to your class/record. I will not describe this feature in this post since you can see the online documentation for it on the Embarcadero Doc Wiki. The part I am interested in is the combination of class constructors and generics. As you might already know generic types aren’t really “run-time entities” but rather “compile-time” ones. This makes the initialization of “static” members of the type a bit more complicated. See for example this record:

type
  TMyType<T> = record
  private class var
    FSomeList: TList<T>;

  public
    ...
    ...
  end;

In this case the static FSomeList variable needs to be initialized to make sense. Normally, in non-generic classes, you would use the “initialization” and “finalization” sections to create and then destroy that variable. In generic classes this is impossible though. You have to get some workarounds (like initializing the list lazily), but you still cannot destroy the list on finalization.

Well, not anymore! Using class constructors and destructors allows you to easily initialize any static member of a type:

type
  TMyType<T> = record
  private class var
    FSomeList: TList<T>;

    class constructor Create;
    class destructor Destroy;
  public
    ...
    ...
  end;

class constructor TMyType<T>.Create;
begin
  // Executed on application initialization.
  FSomeList := TList<T>.Create();
end;

class destructor TMyType<T>.Destroy;
begin
  // Freed on application termination.
  FSomeList.Free;
end;

There is a catch tough. Since generic types are compile-time entities, any unit that uses a specialized generic type basically defines that type in itself. For example if you use TList<String> in four of your units, all four units will internally declare the TList<String> class. This results in the TList<T>’s class constructor to be executed four times – once per unit. This is expected behavior since each type specialization has different static members. For example:

type
  TDistinctType<T> = record
  private class var
    FMarker: Integer;

    class constructor Create;
  end;

If TDistinctType<String> is used in multiple units, each unit’s version has it own FMarker, which means it needs to be initialized for each unit.

The conclusion — be aware that class constructors and destructors for generic types may execute multiple times.

Yes I know I have skipped 0.5. The reality is that 0.5 was due a long time ago, but I did not have enough free time on my hands to complete the unit testing for all new features added. I always try to add as many unit tests as possible to test all possible scenarios.

This release features a general reorganization of some key concepts, new type extension mechanics, boxing, a lot more collections, better support for associative collections through specific Enex extensions … and many more. I am not going to lay out all the changes here since there are quite a few of them. You can view the change log here.

One important note, check out the DeHL.Base.pas unit. There are a number of TOGGLE_XXXXX constants. Toggle them to true or false (if you are using Weaver FT builds) to enable or disable specific functionality. In Tiburon (Delphi 2009) all the features are disabled by default (to be on the safe side).

Get the latest build here. If you stumble upon some bugs please report them here.

Have Fun!

Since some people asked me to improve FSEnum unit (see this post). I decided to give it another go and add some features and improve the others.

So what’s new?

• Optional Boolean parameter for all methods that tells the enumerator whether to return the relative paths (like it did before) or absolute ones.
• There are two levels of “mask matching” (if the enumeration is recursive). The first level allows everything to pass since you need to recurse into all directories. The real mask matching is done at the second level using the TMask (unit Masks).
• Two new methods (overloaded): FileSystemEntries which returns an enumerable over TFileSystemEntry structure.
• A new TFileSystemEntry structure that contains more information about a file/directory.

The next example shows the use of TDirectory.FileSystemEntries method:

var
  Entry: TFileSystemEntry;

begin
  { Show all read-only executable files on drive I: bigger than 1MB }
  for Entry in TDirectory.FileSystemEntries('i:', '*.exe', true) do
    if (Entry.Size > 1000000) and Entry.IsReadOnly then WriteLn(Entry.FullName);

  ReadLn;
end.

The unit is the same (has been updated!)  and can be found here: File system enumeration (702)

In a recent comment on this blog Patrick van Logchem suggested a way of extending an existing object instance with custom data at run-time (originally the idea of Thorsten Engler). The main idea is to be able to “assign” to an arbitrary object (whose sources you cannot change) some other object at run-time. This may prove to be useful in different scenarios when you need some additional data to be carried by an object.

The suggested idea was to use the monitor field which all TObject instances have. If you are not familiar with monitors in Delphi, let me explain: All TObject instances carry a new “hidden” field which is a pointer to a TMonitor (see System unit) structure. This referenced TMonitor structure is actually a kind of synchronization object that lets you solve some common threading tasks easier. You can read more about this in other posts, since this post is about using that field to store data.

The restrictions:

• The solution must not be too simplistic. Simply rewriting the pointer in TObject instances is not allowed.
• Normal monitor routines must function as they did before. This means that we must store a real monitor there alongside with the data.
• No custom functions to simulate monitor support, and no class helpers. See previous item.
• The attached data must be disposed when the object is disposed so to avoid memory and resource leaks.

The features:

• Possibility to extend an object with another object: ExtendObject(Object, Extension);
• Possibility to query an object for an extension: GetObjectExtension(Object): Extension;
• Possibility to remove an extension from an object: RemoveObjectExtension(Object): Extension;
• Object is any type of object in Delphi! No restrictions, no common ancestor; just plain TObject!
• Extension is also simply a TObject value. It’s user-defined in it’s implementation and purpose.

The first implementation I came up with is non-intrusive. I wanted to avoid patching the System exposed functions at run-time. The following list enumerates the design and restrictions of the first implementation:

• The unit must me USED directly after the inclusion of SysUtils in the main source file. This is an inconvenience of course, but it is a required one. Note that the unit must be included AFTER and not BEFORE SysUtils.
• The unit overrides the values in the System.MonitorSupport variable and inserts it’s own custom routines used to obtain and release synchronization objects.
• The unit uses the old System.MonitorSupport routines to do the real job. These routines are normally provided by the SysUtils unit — thus the dependency on SysUtils.
• For each synchronization object requested, my custom routines return a fake handle which is actually a pointer to a structure containing a real handle and a TObject value.
• This method does not use the monitor field per se; rather, it uses a field in the monitor itself.
• Class helpers are used in implementation section to obtain access to internal method in the TMonitor structure.
• While this method is non-intrusive at assembly level, it is surely more complex and uses more CPU cycles.

The second implementation is intrusive! It patches some functions in System unit so that my handler are executed in certain scenarios. The following list enumerates what’s going on in this one:

• The unit should be USED after or before SysUtils. This restriction comes from the fact that monitors initialized before this unit is included have a different format. So there may be (or maybe not) problems.
• Two functions are patched in System unit: TMonitor.Destroy(TObject) and TMonitor.Create(). First one is executed when a monitor is destroyed – normally at object death; and the second one is called when a monitor value is needed for the first time.
• The two injected functions do basically the same thing as the System versions, with a slight turn – a TMonitor + Pointer value is created/destroyed. This bonus Pointer value hold the extension object.
• Class helpers are used internally to gain access to some monitor support routines.
• This method does not incur any overhead on normal monitor operations, so it is the preferred one.

And now to some code:

uses
  SysUtils,
  ObjectExtensions_Intrusive;

type
  TStrExtenstion = class
  end;

  TIntExtension = class
  end;

procedure DoStuff(const A: TObject);
var
  Extension: TObject;
begin
  if A = nil then
    Exit;

  Extension := GetObjectExtension(A);

  if Extension = nil then
    Exit;

  WriteLn(Extension.ClassName);
end;

var
  a, b, c: TObject;
begin
  a := TObject.Create;
  b := TObject.Create;
  c := TObject.Create;

  ExtendObject(a, TStrExtenstion.Create);
  ExtendObject(b, TIntExtension.Create);

  DoStuff(a);
  DoStuff(b);
  DoStuff(c);

  ReadLn;
end.

It’s not hard to imagine that this method can be used is different circumstances – the ideas are all yours!

MEGA WARNING: The attached code is barely tested, possibly unstable and even worse – maybe destructive. This is just a fun and proof of concept code and not something that can be used in applications.

The code can be found in this archive: Object Extensions (375)

Mentioned the authors of the idea.

Me again, and again with enumeration techniques. In this post I will try to coven a very common problem all programmers have to face one time or another: enumerating all files recursively in a directory.

Yesterday I had to do it again, and again following the standard FindFirst … FindNext and FindClose pattern. So I decided to make my life easier and use enumerators for that. Behold the results:

var
  S: String;
begin
  for S in TDirectory.Entries('I:', true) do
    WriteLn(S);
end;

That’s all you have to do to traverse the directory structure for the I:\ drive — simple and clean.

So how does this work and what are the advantages:

• The method Entries is a static method of TDirectory structure. There are two more methods: Files, which returns only files and Directories which returns only directories.
• The exposes static methods return an IEnumerable<String> interface which exposes the GetEnumerator() method. This is an important aspect of the implementation since exposing an interface helps you pass the lifetime management of the enumerable object to the compiler.
• The for .. in loop then extracts a IEnumerator<String> interface and starts iterating over the directory tree.
• The TDirectory.TEnumerator object does not use recursion internally to traverse the tree. It stores the TSearchRec at each level in a TStack<> instance (well it’s kinda like recursion).
• You can “break” off the loop at any moment. Simple and easy.
• It executes more instruction per iteration but I think it’s a manageable trade-off for its ease of use.

Again, th unit can be found here:  File system enumeration (702).

Have fun!

I have decided to extract the BigInteger and BigCardinal data types from DeHL and make a bundle with them separately. Someone may be interested in these types and not so much in DeHL. The attached unit should compile on D2007 and D2006 (not sure for earlier versions).

What can you learn:

• Shows you how to develop “value types” with predictable states in Delphi using records, operator overloading and “managed types” (in this case a dynamic array).
• How to develop a proxy class that plugs your type into the Variant manager.
• Demonstrates how to use operator overloading (and I have overloaded a lot of operators there).
• The reliance of the predictable state of the managed types to check whether a value is initialized or not.
• Shows how the use an in-place BCD-izer algorithm that helps converting a big number into a string. It took me a quite some time and tweaking to come up with it so it may be interesting to someone.

Where can you use it:

• Everywhere, except FOR loops of course.
• The types are transparent and should be used the exact way as normal integer do.
• Use it in applications that require very big integer values.
• … you’ll come up with something else.

A small example of course:

var
  a, b: BigInteger;
begin
  a := StrToBigInteger('8942374632463284632623837643468589' +
        '26367423509458904389575645349857');
  b := a * a;
  WriteLn(IntToStr(b));
end.

… or as variants …

var
  a, b: Variant;
begin
  a := StrToBigInteger('8942374632463284632623837643468589' +
        '26367423509458904389575645349857');
  b := a * a;
  WriteLn(b);
end.

Final note: the code is tested since it’s a part of DeHL (in which I try to test everything I can) so it’s safe to use. BUT if you find any bugs, please drop me a line.

Finally! The code! You can find it here: Arbitrary precision Integers (339)

I was playing around with some (default Windows) programs these days and noticed that taskbar items for those programs show some cool effects whenever there is some progress going on So I decided to write a little unit to take advantage of these effects in one of my Delphi applications.

To take advantage of the new Windows 7 TaskBar API take a look at the ITaskBarList3 interface (an probably 4 for that matter). I have translated the interface to Delphi and added some initialization code and came up with a little unit that exposes a class helper for TCustomForm.

The final user-friendly code looks like this:

procedure TForm1.Button1Click(Sender: TObject);
begin
  SetProgressState(tbpsError);
  SetProgressValue(50, 100);
end;

The SetProgressState and SetProgressValue methods are inserted by the class helper. The example makes the taskbar icon for the window look like a 50% complete red progress bar (indicating an error):

or something like this for a “indefinite” state (of course the animation is missing so you can’t see the effects):

If you want to play with the unit, take a look here: Windows 7 TaskBar helper for Delphi (644)