Launch

We’re finally at version one point zero.
And the new website is also finally up, which has its own blog now so I’ll be blogging on there from now on.

Check out www.metadrone.com

About lexing

Earlier I wrote about having issues with regular expressions for lexing and parsing.

Well my suspicions have been confirmed after reading Rob Pike’s take on using regular expressions for this purpose: http://commandcenter.blogspot.com/2011/08/regular-expressions-in-lexing-and.html

 

This doesn’t mean I won’t look further into seeing how to use regex’s for parsing the syntax down the track. But I am more than happy with the performance of just plain old looping over characters.

 

 

 

 

 

 

 

Start-up Obstacles

It’s been a while I know.

There is still progress, and all will be revealed in the launch. However there have been set backs: a contracting vortex (keeping some clients happy and completing stuff), being a dad for the first time (18 June – it’s a girl!) and taking some time off for it, and also the temptation to pivot because I feel this project could commercially be a dud.

So while I have contemplated open sourcing Metadrone, I will continue with my vision and see where it leads. Besides, I’m a little self conscious with my coding.

I’ve put off some new features I have thought about until another iteration, and will just deliver version one and see what I can fit in with feedback – if I get any.

 

The boring stuff

In the pipeline now is the new website.

It’s a whole heap of work which can be easy to belittle and delay while you tackle the “real” work – the fun stuff – that is the programming. I’m tempted to outsource for the design, CMS set-up, etc. But in all reality it’s not that difficult, just really time-consuming. Additionally I want to D.I.Y. to appreciate the complete experience of this bootstrappin’ startup venture.

I’ve decided to use WordPress, I think I’m getting the hang of it, and there’s a ton of design templates out there for it. Most of the basic pages are up: download links, about screen, contact, blog, etc. What’s left is me retreating back into my cave to tackle two big jobs:

1. Documentation.
   I think the most value would be tutorial examples rather than dry syntax reference. Example code always seems more helpful, and the language syntax is not that complicated anyway. The reference will of course still be there, but it will definitely be not very wordy and illustrated with examples.
2. Community.
   This will be a difficult challenge and a “project phase” in itself. I have given it some thought, but not in a great deal of detail and there’s still some questions to answer for myself. For instance, should the forum functionality be part of the community pages. And if so, can a wordpress forum plugin be customised easily to work with other things specific to Metadrone. How can I use the user accounts and link them with the Metadrone application. Does this mean web service access to the wordpress mysql database.

Right now, I’m setting up the documentation. I have to tweak a template for it and then commit myself with steely determination to complete the reference and tutorials. Every programmer will tell you of the boring, soul-numbing torture that is documentation. But it must be done. And done well.

I’ll complete the community aspect next.

And THEN, it’s marketing and branding to get splash screens designed and the like.

And then launch.

Hopefully.

Surgery

Just a quick update.

In the spirit of the “iterate early and iterate often” mantra, I had to go back and clean up some shortcuts I had taken to get some features out the door and tested and used. Most of these tidy ups would not be noticeable performance-wise. I just tied up some loose ends in the code parsing and did some refactoring and removed duplicated code.

A bonus of this would be quicker delivery of new language features, and the directives (or “main” as I think I will rename it to) uses the same compilation/parsing routine as the templates; meaning code-completion in the main(), looping and conditional constructs in the main(), and you cal call templates within templates for recursive, stack-blowing goodness. An example of why you would want this would be for when you wish to loop through tables linked to a current table via foreign key constraints. These loops will be provided soon – including loops through indexes (and columns within indexes). And even further down the track loop through model entities themselves, not just a database schema.

This lead to the inevitable implementation of a symbol table for variables. Hopefully this will better support the code-completion which still has some niggles needing to be ironed out.

It was a surgical week. I don’t really like them. It’s that long stretch in hacker desert where the guts have been ripped open and tossed about all over the place .My AnkhSVN plug-in glowing red boxes in the solution explorer all over like a rash. But finally I reached that oasis where I can hit the commit button.

Performance improvements have been negligible. It’s plateaued so far. Compilation speed isn’t too bad, at a rate of about 5,000 lines per second. The syntax is pretty simple, it could be improved if I was brilliant but there probably won’t need to be any for the near future. Unless someone comes up with a chunky template of 40k lines. But I think the main concern from there would be generated output.

So generated output averages around 260k of data per second, or 3.84 seconds per megabyte of data . To test this I ran a template to generate typed datasets on a database with 1733 stored procedures and that took 34 seconds on average resulting in an 8.85mb code file. (I’d hate to think about the agony of dragging and dropping those 1733 stored procedures from the server tab into the dataset designer).

The performance will do for now. There is one more trick not available to the outside world and that is to not obfuscate the executable. A vanilla build took 22 seconds on average to generate the same 8.85mb of data; approx 402k per sec, or 2.5 secs for mb. That’s quite a performance hit with obfuscation, about 35% slower. I don’t know whether it’s the obfuscater itself, or actual obfuscation. Open source it you cry? One day perhaps. Depending of course on how clever (or successful) my revenue strategy is for this project.

The real solution I guess would be to generate managed code as part of the compilation process and compile and run that actual code. The Metadrone syntax will essentially then be a kind of a macro to create c#/vb code generation code (or a T4 template).

Beta 0.11

Beta 0.11 has been released.

The new features are:

1. CodeDOM implementation.
2. Expression evaluations have been revised and are much more robust and intuitive. This forms a kind of bedrock for the language.
3. Code completion fixes.
4. Searching has been revised and works more predictably.

There are some remaining features I have mentioned previously that will be implemented in the 1.0 launch. The next phase is now the launch with hopefully a stable and fully functional and feature rich code generation tool. Accompanying this will be the revamped website, documentation, and community facility.

Once we’re launched, we’ll try our luck with diving into the mysterious void of marketing and promotion. We’ll really find out if Metadrone is actually “wanted” by the developer community.

Temptation

I have been swimming in recursion soup the past fortnight to improve expression evaluations. For a moment I thought it would be “smarter” to build VB/C# code in codeDOM representing an expression in a template and substituting the primitives (variables) on the fly. This would mean having actual expressions compiled using the .NET framework and identified in a symbol table, and when used in the template the code is referenced and the variable values passed into their placeholders. This would probably would have resulted in a different type of spaghetti and be restricted to a language designed for programming, not model-driven code generation.

So it was head down, bum up, and(((lots of these) to sort), out) and precedences^to painfully(figure, out).

The hard work has paid off, and the result has been surprisingly neat. Also the syntax is not therefore bound to VB/C# to improve portability if need be.

That being said executing VB/C# code is ready to go. Various other bugs have been fixed and optimisations have been made to the compilation process (the tokenisation, building the syntax tree, etc). Text searching has been fixed up properly, and the code-completion has had some more annoying features done away with.

There are some more things to add onto for the final polish up before launch but these fixes are important enough for another beta release, which will come very soon.

CodeDOM

Adding .net code execution wasn’t even 1% as painful as I was fearing it to be. I suspect this is how the more established player(s) carry out code generation: codesmith, mygeneration, even this fellow; and then they built upon this approach.

Using dot net code (or T4) has some pros and cons.
Pros:

1. Low learning curve with using an existing language syntax.
2. A very much shorter implementation time because a language base is already there (codeDOM implementation took me only a few days total).
3. Integration with existing libraries, leveraging more power, saving more time.

Cons:

1. Extra dependencies such as the framework or run-time.
2. In my honest opinion imperative, object oriented, procedural languages are a little cumbersome for use with this type of code generation. The resulting template code ends up performing some gymnastics and becomes difficult to read. And annotations galore!

I think the complexity and readability issue weighed the heaviest for me. From my experience the intricacy and power of established langues is best hidden from the “main meat” of code generation. The Metadrone syntax is intended to be very lean. There’s no real need for object orientation, type safety, inheritance, overloading, etc. It’s the old 80/20 rule. 80% would be basic looping through model entities – which currently are database tables and columns – but 20% may require some clever thinking which is where using a mature and sophisticated programming language would be useful. Rather than expand upon the Metadrone syntax, which will just make life harder for myself in development effort and for the end user in learning new syntax, Metadrone can just pass information into managed code and get the result back.

To define the managed code you can add code items as you would a template. You can either add a VB or a C# code “file”. You can add as many as you want. The parser will just join them all together appropriately (all the VB code and C# code separately) and use the codeDOM library to compile the code. Instantiating and invoking is performed with the new function “runvb” or “runcs”.

For example if you had some C# code:

public class CS    {
        public string Blah(string blah)
       {
            return blah + " blah";
        }
}

within Metadrone you can invoke it:

<<!for table tbl in MainConn!>>
<<!for column col in tbl!>>
<<!tbl!>> - <<!runcs("CS.Blah(""some text""" + col + ")")!>>
<<!end!>>
<<!end!>>

I intend to make Metadrone objects properly accessible within managed code. Therefore if you wanted, you could just use plain codeDOM instead of the template language. The flexibility will be there depending on need and preference.

Beta 0.10

A quick fix for a potential issue the boys at Agility discovered.

The default SQL metadata query can sometimes not pick up a primary key if  it has other constraints against the column. If you are familiar with the default query it just gets a top 1 on the constraints linked to the column, so if there are multiple constraints you aren’t necessarily going to pick up the “PRIMARY KEY” constraint all the time.

For now I’ve limited the constraints to being “IN (‘PRIMARY KEY’, ‘FOREIGN KEY’)”. The compatibility checker will modify the particular query for you when you load an older version.

The issue with multiple constraints has got me thinking, what about if you want to get just indexes? <<!for idxcolumn col in tablevar!>>

Another little nugget to implement in there.

Anyway, there were a couple of other little issues:

• The “adduseonly” and “addignore” functions weren’t behaving correctly.
• It would have been nice to indicate “project/folder/paths” for packages within any project folders. So this was put in there. Nice.

What’s for Beta 0.11? Well hopefully:

1. CodeDOM implementation. Call .NET code from within a template.
2. Execute a SQL script against a source.
3. It would be really frikken handy to enable repositioning of tabs (grab and slide).
4. Variables and probably templates declared against a symbol table.
5. Another nice feature <<!for table tbl in table!>>. Meaning loop through tables linked to a parent table by foreign key.
6. More code completion tweaks and enhancements.

After this we should be good to go for version 1.0. Promise!

Beta 0.9

Changes for this release:

• Transformations added.
• Console debugging window added.
• Parsing optimisations.
• Pre-compilation has been disabled since the optimisations are fast enough to not need it.
• Some more eye-candy and general “polishing”.
• Many bug fixes, and the code completion has been improved and is less annoying.

Transformations are applied for each row of meta-data, and defaults are added to the SQL Server, Oracle, and Access source providers.

A symbol table implementation has not been performed as of yet. It is quite an invasive piece of work to do, so it will have to wait until another iteration. Also, I was hoping to enable codeDOM execution and direct script execution against a database connection, but again this will best best applied in a future iteration. The current changes need time to settle.

Documentation will be up soon, it is the next priority.