DaggerTech.Data finally converted

It has been a long road converting my C# ORM to Go, but it is finally done (although I do need to clean up the code)
Unfortunately, my day job got in the way of working on this, but I still gotta eat.
Anyway, the ORM (called GoDagger), like DaggerTech.Data works by creating the tables when it needs them, so no initial setup required. This does mean that any application using it can start up faster, but it can have a minimal performance hit in the early stages of execution.
The programmer is completely shielded from the SQL of the underlying database, allowing him, or her, to focus directly on the logic.
Each table is given 4 fields:
        ID                         VARCHAR(36) NOT NULL PRIMARY KEY
        CreateDate         BIGINT NOT NULL
        LastUpdate         BIGINT NOT NULL
        DeleteDate        BIGINT NULL
As you can see, I am using numeric values for dates. This allows me to port GoDagger to other databases without the need for any other conversion of date values.
The Model struct in Go to match to this is as follows:
type Model struct {
        ID                        *string
        CreateDate        time.Time
        LastUpdate        time.Time
        DeleteDate        *time.Time
}
As you can see in Model, both ID and DeleteDate are pointers. This allows me to have nil values in those fields. We use a nil value in ID to indicate if the record has ever been saved and the DeleteDate is nil if the record has not been deleted. The reason for the DeleteDate is that in some systems, it is better to disable a record rather than delete it. By default, GoDagger will not retrieve any record that has a DeleteDate, but this can be overridden.
To create a new model, we use composition of the Model struct, plus the use of struct tags:
type TestModel struct {
        godagger.Model
        Age         int                 `dagger:""`
        Name         string         `dagger:"size:20"`
        Nickname        *int                 `dagger:”size:20”`
}
For this familiar with GORM, you will see a resemblance. This is purely coincidental as I had not looked at GORM until a few days ago. The big difference, though, is GORM will include fields by default, unless specified to be ignored. GoDagger, on the other hand, will only include fields with a ‘dagger’ struct tag.
This is a very lightweight ORM as it does not create links between models for you. That is up to the developer. This is how DaggerTech.Data works as a lot of the data is to be passed as JSON to web applications and if I were to allow automatic linking and loading, then there would be the potential for massive amounts of data to be passed, as well as infinite loops.
To save a model to the database, it is simply a case of calling the Save function in GoDagger. This will look at the ID field and determine if the record needs to be created or updated.
tm1 := &TestModel{
        Name: "Test1",
        Age: 42,
}
godagger.Save(tm1)
Note that the save method requires a pointer, as do all the functions in GoDagger.
Searching, deletion, and some other helper functions are also available, which I will detail in a future post.

OOP & Reflection in Go??

Converting DaggerTech.Data to Go is proving to be more of a challenge that I realised.

After over twenty years of object oriented programming, I am having to unlearn a great deal to work the Go way. Whilst Go does have a form of object orientation, to do what the ORM library needs to do, I need to re-work a lot of the code to a non-OOP fashion.

The main issue I am facing, is that I am so used to inheritance. The ability to call a method of a base class, with that base method being able to view the properties of the derived class (when using reflection). Unfortunately, this is not possible with Go’s method of embedding structs. The answer is it use interfaces and have functions that do not have a receiver, but the interface as the parameter. Not sure if that makes sense, but I will illustrate:

The following code is how I would have performed the reflection in a normal OOP manner. In fact, DaggerTech.Data does this, but in the C# version:

package main

import (

    "fmt"

    "reflect"

)

type Base struct {

    Name string

}

type Derived struct {

    Base

    Age int

}

func (b Base) DoReflection() {

    t := reflect.TypeOf(b)

    for i := 0; i < t.NumField(); i++ {

        fmt.Println(t.Field(i).Name)

    }

}

func main() {

    b := Base{}

    b.DoReflection() // OK. Displays the field Name

    d := Derived{}

    d.DoReflection() // Not OK. Only displays the field Name

}

Result:

Base Name

Base Name

The problem with this is that the call to DoReflection is only dealing with the Base struct, even though the DoReflection is elevated when the Derived struct embeds it. What is actually happening, is the compiler sees d.DoReflection() and actually replaces it with d.Base.DoReflection(), therefore passing the Base as the receiver.

To get around this, we need to make the DoReflection method a function receiving an interface:

package main

package main

import (

    "fmt"

    "reflect"

)

type Base struct {

    Name string

}

type Derived struct {

    Base

    Age int

}

func DoReflection(b interface{}) {

    t := reflect.TypeOf(b)

    for i := 0; i < t.NumField(); i++ {

        fmt.Println(t.Name(), t.Field(i).Name)

    }

}

func main() {

    b := Base{}

    DoReflection(b) // OK. Displays the field Name

    d := Derived{}

    DoReflection(d) // Not OK. Displays Base and Age

}

Result:

Base Name

Derived Base

Derived Age

But now we have a different issue. When the Derived struct is inspected, it sees the Base struct and just reports that, not the ‘inherited’ fields. For this we need to employ a little recursion:

package main

import (

    "fmt"

    "reflect"

)

type Base struct {

    Name string

}

type Derived struct {

    Base

    Age int

}

func DoReflection(b interface{}) {

    t := reflect.TypeOf(b)

    v := reflect.ValueOf(b)

    for i := 0; i < t.NumField(); i++ {

        if v.Field(i).Kind() == reflect.Struct {

            DoReflection(v.Field(i).Interface())

        } else {

            fmt.Println(t.Name(), t.Field(i).Name)

        }

    }

}

func main() {

    b := Base{}

    DoReflection(b) // OK. Displays the field Name

    fmt.Println("") // Added an empty line for readability in the results

    d := Derived{}

    DoReflection(d) // OK. Displays Name and Age

}

Result:

Base Name

Base Name

Derived Age

As you can see in the results, we now have both of the fields from the Derived struct. As a bonus, we can even see in which struct the field was declared.

In the database library, I need to also get the field type and any tags that are applied to the fields, but that is trivial compared to re-working my brain for this. I also need to put in special conditioning for the Time struct (equivalent to the C# DateTime).

New Project

Time to start a new project.

This one, without giving too much away, is a large scale project, for medium to ‘semi’-large enterprises.

It will be web based, allowing for a large number of simultaneous users, as well as being driven by an SQL database.

Having a number of certifications and qualifications in various languages, I have the opportunity to choose my development stream.

Firstly, I considered using Xojo, but the problem there is that my licence for Xojo has expired and as I am now using macOS Catalina, I can no longer use 32 bit applications. I also don’t really want to pay for the licence when there are free options available, so it is time for me to move on.

Next, we have PHP. I have a lot of experience with PHP and have developed a large number of web sites and applications using it. However, being a scripted language, it has inherent problems, primarily speed and the potential for vulnerabilities. On the plus side, PHP has a very large community and a vast number of libraries to call upon, but, it does need a web server and runtime to be deployed, before deploying the application.

My day job requires me to develop software written in dotnet, or more specifically, C#. This would fix the speed issue (to a point), and, if I use dotnet core, I wouldn’t need to pre-deploy a web server. Through my day job, I do have a certification from Microsoft for developing applications in C#, but there are a number of issues, primarily speed and the need to deploy the dotnet runtime, which is quite large, on the target system.

Each of the options mentioned above are available on all of the three main platforms: Windows, Linux and macOS. The next option is not available on Windows, which is why I have rejected this one also. Swift is known for being the primary language for creating iOS and macOS software, however, as it is now open source, it is just as easy to use it for Linux, and with some jiggery-pokery, Windows (but this is not straight forward). There are three main frameworks for creating web applications with Swift, but it feels like a bolt on, rather than a native solution.

This brings me to the final, and selected, solution: Go. I recently completed a course in development with Go, which doesn’t make me anything like an expert, but I feel I have enough understanding of the core syntax and key concepts to be able to utilise it for a production project. Go produces native binaries, so no runtimes required. No web server is required either as, like dotnet core, the product is an application that acts as the server as well. Finally, Go, being a new language, is designed for multi-core systems, this means that concurrent programming is the norm for it (Rust is similar in that respect, but I prefer the syntax of Go). On the down side, it is not a strictly object oriented language (although some would argue that the syntax allows for a form of OOP). This means that I need to learn ta new way to design and develop he software.

The first task at hand is to convert my database ORM package. About a year ago, I wrote a library for C#, called DaggerTech.Data which divorces nearly all of the database functionality from your application code, allowing you to focus on the application. The library was designed for use with Microsoft SQL Server.

I have also ported DaggerTech.Data to PHP, although this one is designed for MySQL and MariaDB.

For this project, I will be using Postgres as the database. I want to keep the costs to the customers down, so by using a truly open source database, it ticks all the boxes. SO it does mean that I need to rewrite DaggerTech.Data in Go (currently codenamed GoDagger), which is a bit of a challenge as the original was completely object oriented and it uses reflection to provide the functionality, but the documentation for Go is very comprehensive, so it shouldn’t be too bad.

Anyway, just an update to let you know what I’m up to these days. I will keep you updated with developments.

Watching A Folder pt 2

A quick revisit to the Folder Watcher.

Due to the way Xojo handles threads, it appears that there is no need to yield the thread explicitly within this class.

I believe, but I may be wrong, this is because Xojo has the intelligence enough to yield to the next thread automatically when a loop is encountered. To be honest, I’m not sure, but I ran a test with the yield command commented out, and I experienced no lock up, the application remained responsive.

I have heard, from the Xojo forums, that Xojo does not use pre-emptive threads, so they have to explicitly cooperate with the other threads, but it appears that there is some contingency already coded into the Xojo language for this when using loops.

Plugin Vendors

This is bit of a rant, but I will try to be polite.

I have noticed in a number of developer forums that certain people seem to treat them as a sales market for their own products.

I have no problem with people touting their stuff, hell, we all do it. However, when I post on a forum that I am not interested in plugins, or I have created a piece of code so that I do not need plugins, I do find it frustrating when vendors say “My plugin can do that” or “You can achieve the same thing with XXX plugin”.

I know that some people rely on plugins, and the vendors of these plugins have to make a living, but, for the record, unless I explicitly state “I am looking for a plugin for XXX”, then please assume that I do not want to hear about your plugin.

There are reasons why I do not use them:

1. I am on a very tight budget and cannot afford them
2. I enjoy tinkering with code and seeing what I can get XXX language to do
3. I am a programmer… if I need a plugin, I will bloody well write it myself

Watching A Folder

Been a while since I published some (hopefully) useful Xojo code, so this time I have quite an interesting one.

There are times when it is necessary to get your application to respond to changes in the file system. This is often used for programmers tools where a directory listing is used rather than storing each of the files in the project file, or if your application needs to process data sent via FTP. There are many other scenarios where this feature could be equally useful, but these are the only two I have had to address my self, so far.

In the .Net world, we have the FileSystemWatcher class that can accomplish this for us quite easily. Unfortunately in Xojo, we do not, so we need to roll our own.

The first thought behind this is to simply create a Timer that periodically checks the folder for changes.

OK. But there are some issues with this:

1. How does the Timer event know if the folder has been altered?
2. What happens with a particularly large folder? Just using the Time Action event would hijack the main thread and cause our application to become unresponsive when the checking is being done.
3. What happens if a subfolder is amended?

There are other potential issues, but these are the main three.

My solution is to create a new class, that I will call FolderWatcher.

Class FolderWatcher

End Class

This class does not need to inherit anything, nor use any interfaces, so simply select the Class option from the Insert menu.

When creating a class that fires events, as this will, I like to set up the events definitions at the start, so right now selective Event Definition option from the Insert Menu and name the definition Action. This doesn’t need any parameters as the FolderItem will be retrievable from the class instance.

Talking of the FolderItem, we ought to set up that property now. From the Insert menu, click Property. Call this property mFolder, set the Type to FolderItem and its scope should be Private.

As I said before, we will need to retrieve the FolderItem during the Action event, so add a Computed Property, called Folder, keep this property Public so we can access it from outside the FolderWatcher instance. In the Get method of the Folder property, simply return the mFolder private property

.

I said before that using a Timer to check the folder would be inefficient and cause our application to lock up. However, we do need a Timer to trigger the file check and to see if the flag (which we will set up in a minute), to indicate the folder has changed, has been set.

So now we create the Timer property of the FolderWatcher. For simplicity sake, let us just call this mTimer and set it’s scope to Private.

We will also need two private flags within the class. So, set up two private Boolean properties (I’m sure you have worked out how to create properties by now) and call them mEnabled and mFolderChanged. Set both of the default values of these properties to False.

We only need to expose the mEnabled property to outside the class, so we do this with a computed Property called Enabled. This is also a boolean property. The Get method of this property simply returns the value of the mEnabled private property. However, the Set method sets the value of mEnabled, but also sets the Enabled property of the mTimer property.

  mEnabled = value

  mTimer.Enabled = value

There are just two more properties to add, so we may as well do that now, before we move on to the methods.

Add a private property called mThread. This is of type Thread and will be used to perform the check of the folder without interrupting the main thread.

Add another private property called mFolderDetails and set this to a String. This will store the structure of the folder for comparison during the execution of mThread.

Where have we got too so far?

By now, you should have a class that looks like this:

If you were to try to use this class right not, it would do nothing. Now we need to add the functionality with methods.

First up, the all important constructor.

Create the constructor like any other method. Select the Method option from the Insert menu.

Name this method Constructor and ensure its scope is set to Public. We also need to pass two parameters to the constructor: A folderItem, we will call Folder and a Boolean we will call AutoStart.

The constructor is where we will set everything up.

Firstly, check that we have actually passed a Folder, and not a file to the constructor. This is simply a case of testing the Directory property of the Folder variable. If this returns False, raise a runtime exception.

Now we have the preliminary check out of the way, we can continue with the set up:

Store the Folder in the classes mFolder property.

Create an instance of the Thread class and configure it. The only configuration we need to do here is set the event handler (which we will create in a moment).

Create an instance of the Timer class, and configure that also. This requires a little more configuration. We set the event handler (again, configured in a moment), the period between each trigger of the Timer Action event, the Mode property of the Time, and whether or not we want the Timer to start immediately.

We should end up with code like this in the constructor:

' This class only watches folders, not individual items, so if the

' item passed is not a folder/directory, raise an exception

If Not Folder.Directory Then Raise New RuntimeException

' Store the folder

mFolder = Folder

' Create a new Thread object and attach the Run handler

mThread = New Thread()

AddHandler mThread.Run, AddressOf ThreadAction

' Create the Timer object and add the Action handler

mTimer = New Timer()

AddHandler mTimer.Action, AddressOf TimerAction

' For now, set this Timer to fire every second

mTimer.Period = 1000

mTimer.Mode = Timer.ModeMultiple

' Enable the Timer if AutoStart os True

mTimer.Enabled = AutoStart

The first thing we need to address (no pun intended) is creating the event handlers for the Thread and the Timer. These are the key methods that allow this class to perform its desired task.

To create the event handler for the Thread, we simply create a new method, as we did for the constructor. This time, we will call the method ThreadAction, we set it to Private and have a single parameter called Sender of type Thread.

This method does not return a value.

The body of this method gets the structure of the folder and compares it to the stored structure. If there is a difference, then the mFolderChanged flag is set to True.

' When the Thread runs, get the string value of the folder structure

Dim NewDetails As String = GetFolderDetails(mFolder)

' If the structure has changed since the last check (or this is the first check)

' update the stored details and set the folder changed flag to true

If NewDetails <> mFolderDetails Then

mFolderDetails = NewDetails

mFolderChanged = True

End If

That is all there is to it. The main work of the thread is actually performed in the method GetFolderDetails, but we will get to that after setting up the event handler for the Timer.

To create the event handler for the Timer, we simply create a method in the class. As we can see in the AddHandler call for mTimer, we need to call this method TimerAction. So, that’s what we will do. Repeat the steps to create the ThreadAction method, but this time, call it TimerAction and have the parameter be of type Timer:

The body of the TimerAction method is just as simple as that of the ThreadAction:

Check the FolderWatcher is enabled, if not, exit the method immediately. If the mFolderChanged flag is set to True, reset it to False and then raise the Action event of our FolderWatcher class. Finally, if mThread is not already running, call the Run method of mThread:

' If the file watcher is not enabled, return from this handler

if Not mEnabled Then Return

' If the folder changed flag is set to true, 

' reset it to false and raise the FolderWatcher.Action event handler

If mFolderChanged Then

mFolderChanged = False

RaiseEvent Action()

End If

' If the thread not is running, call its Run method now

If mThread.State <> Thread.Running Then mThread.Run()

The GetFolderDetails method is the main work horse of the class, this is created as any other method, but this time, we pass a FolderItem as the only parameter and we return a string containing the structure of the folder.

As this is more complex that the other methods, we will go through this in closer detail:

Dim FolderDetails() As String ' An array of the folder contents

Dim Index As Integer = 0 ' The current index of the folder’s children

Dim YieldCount As Integer = 10 ' This is used to reduce the amount of yields

These three variables are initialised at the beginning. The first two should be self explanatory, however, the third will require some explanation.

When we execute a separate thread, we need to periodically yield control back to the main thread to prevent the application locking up. This is particularly important in long running, or potentially long running, threads. From my experience, the standard practice for this is to yield control at the end of each iteration of a loop. However, this can effectively cause the thread to be much slower as the ‘behind the scenes’ code to yield the thread builds up. To prevent this, I have found a yield counter is useful. This decrements down to zero on each iteration, when it hits zero, it yields the execution to the main thread. Depending on the complexity of your thread, you may want the yield counter to start at a higher value (say, 100), or lower, if the loop is particularly complex (possibly 2). 

We can now examine each child of the FolderItem passed to the method. This is done with a simple While loop, existing when the Index is equal to, or greater than the number of children.

 

' Get each child of the folder

While Index < Folder.Count

As this example is using the classic framework, the child items of a FolderItem are number from 1, not 0. So, we increment the Index at the start of the loop.

' Using the Classic Framework, so the FolderItem 

' children are in a 1-based array. Therefore,

' the increment is done at the beginning of the loop

Index = Index + 1

  

Get the name and modification date of the child and create a string based upon this details. In this example, I use the TotalSeconds property of the modification date. Please note, in a real world example, it may also be an idea to get the file size and add that to the string, but we won’t worry for this example.

' Get the name and modification date of the child FolderItem

' (NB: Should also be checking file size)

Dim ChildFolder As FolderItem = Folder.Item(Index)

Dim ChildName As String = ChildFolder.Name + " : " + _

Str(ChildFolder.ModificationDate.TotalSeconds)

  

If this child is a folder, we add a marker to the beginning of the string (in this case , I add “[D] “ and append the contents of that folder. TO get the contents of the folder, simply call this method recursively, passing the child as the FolderItem.

' If the FolderItem is a directory, then Add [D] 

' to the beginning of the name and append the

' structure of that folder by recursively calling this method

If ChildFolder.Directory Then ChildName = “[D] " + ChildName + _

GetFolderDetails(ChildFolder)

Add the final string for the child details to the array of strings that give us our structure .

' Add the details of the FolderItem to the array of details

FolderDetails.Append(ChildName)

Now we do the YieldCounter processing described earlier.

' Decrement the YieldCount

YieldCount = YieldCount - 1

' If the YieldCount is at zero, yield to the next thread 

' to prevent application lock up

If YieldCount = 0 Then

App.YieldToNextThread()

  

' Reset the YieldCount

YieldCount = 10

End If

Wend

Finally, we return the array of strings as a single string.

' Return the details as a single string

Return Join(FolderDetails,"|")

Now, to finish the class, we need to add a Destructor and two convenience methods.

The Destructor is required as shutting down your application with an active Timer or Thread may cause it to crash, and shutdown ungracefully. We want to avoid this for several reasons, including the fact that it doesn’t look very professional if your crashes every time your quit.

A Destructor is created in the same was as a Constructor, however, it does not, and cannot, take any parameters. To create it, just create a method called Destructor with a Public scope:

All this needs to do is call the Stop method of the FolderWatcher:

' Stop the FolderWatcher to prevent errors on application close

Stop()

However, we don’t have a Stop method, se we will create it now.

The Stop method is simply a convenience method that sets the Enabled property to False.

Create this by creating a public method called Stop, that takes no parameters, and give it the following body:

' Stop the FolderWatcher Timer

Enabled = False

To compliment this, we can create a Watch method that does the inverse, setting the Enabled parameter too True.

Create this by creating a public method called Watch, that takes no parameters, and give it the following body:

' Start the File Watcher

Enabled = True

And that is the FolderWatcher class, we should now have a class that looks like this:

Class FolderWatcher

Event Action()

Sub Constructor(Folder As FolderItem, AutoStart As Boolean = False) 

' This class only watches folders, not individual items, so if the

' item passed is not a folder/directory, raise an exception

If Not Folder.Directory Then Raise New RuntimeException

' Store the folder 

mFolder = Folder

' Create a new Thread object and attach the Run handler 

mThread = New Thread()

AddHandler mThread.Run, AddressOf ThreadAction

' Create the Timer object and add the Action handler 

mTimer = New Timer()

AddHandler mTimer.Action, AddressOf TimerAction

' For now, set this Timer to fire every second 

mTimer.Period = 1000

mTimer.Mode = Timer.ModeMultiple

' Enable the Timer if AutoStart is True 

Enabled = AutoStart

End Sub

Sub Destructor()

' Stop the FolderWatcher to prevent errors on application close 

Stop()

End Sub

Private Function GetFolderDetails(Folder As FolderItem) As String

Dim FolderDetails() As String ' An array of the folder contents

Dim Index As Integer = 0 ' The current index of the folderʼs children

Dim YieldCount As Integer = 10 ' This is used to reduce the amount of yields

' Get each child of the folder 

While Index < Folder.Count

' Using the Classic Framework, so the FolderItem

' children are in a 1-based array. Therefore,

' the increment is done at the beginning of the loop 

Index = Index + 1

' Get the name and modification date of the child FolderItem ' (NB: Should also be checking file size)

Dim ChildFolder As FolderItem = Folder.Item(Index)

Dim ChildName As String = ChildFolder.Name + " : " + _

Str(ChildFolder.ModificationDate.TotalSeconds)

' If the FolderItem is a directory, then Add [D]

' to the beginning of the name and append the

' structure of that folder by recursively calling this method

If ChildFolder.Directory Then ChildName = "[D] " + ChildName + _

GetFolderDetails(ChildFolder)

' Add the details of the FolderItem to the array of details 

FolderDetails.Append(ChildName)

' Decrement the YieldCount 

YieldCount = YieldCount - 1

' If the YieldCount is at zero, yield to the next thread 

' to prevent application lock up

If YieldCount = 0 Then

App.YieldToNextThread()

' Reset the YieldCount

YieldCount = 10

End If

Wend

' Return the details as a single string

Return Join(FolderDetails,"|")

End Function

Sub Stop()

' Stop the FolderWatcher Timer 

Enabled = False

End Sub

Private Sub ThreadAction(Sender As Thread)

' When the Thread runs, get the string value of the folder structure 

Dim NewDetails As String = GetFolderDetails(mFolder)

' If the structure has changed since the last check (or this is the first check) 

' update the stored details and set the folder changed flag to true

If NewDetails <> mFolderDetails Then

mFolderDetails = NewDetails

mFolderChanged = True 

End If

End Sub

Private Sub TimerAction(Sender As Timer)

' If the file watcher is not enabled, return from this handler 

If Not mEnabled Then Return

' If the folder changed flag is set to true, reset it to false and raise the FolderWatcher.Action event handler

If mFolderChanged Then 

mFolderChanged = False 

RaiseEvent Action()

End If

' If the thread not is running, call its Run method now

If mThread.State <> Thread.Running Then mThread.Run()

End Sub

Sub Watch()

' Start the File Watcher 

Enabled = True

End Sub

Enabled As Boolean

Get

return mEnabled

End Get

Set

mEnabled = value

mTimer.Enabled = value

if mThread.State = Thread.Running Then mThread.Kill()

End Set 

End Property

Folder As FolderItem

Get

Return mFolder

End Get 

End Property

mEnabled As Boolean = False 

mFolder As FolderItem 

mFolderChanged As Boolean = False 

mFolderDetails As String

mThread As Thread

mTimer As Timer 

End Class

So, after all that, how do we use the FolderWatcher class?

Add a FolderWatcher property to your application

Create an instance with the folder you wish to watch

Add an event handler

Respond to the event handler

Or in a little more detail, If we are adding the FolderWatcher to our main window to watch the Desktop folder:

Create a new property of type FolderWatcher

In the Open event of your Window, create an instance of FolderWatcher, passing the folder you wish to watch

Add an event handler to the Action event of the FolderWatcher

Call the Watch method of the FolderWatcher

mFolderWatcher = New FolderWatcher(SpecialFolder.Desktop)

AddHandler mFolderWatcher.Action, AddressOf FolderWatcherAction

mFolderWatcher.Watch()

I have provided a download to a small demonstration application that watches your desktop. Run it and then alter your files on the desktop and it should respond.