Vice City: Multiplayer

I am reserving this topic for a plugin that I'm working on. It's an alternate version of the Squirrel plugin designed for advanced scripts and script-writers. The source isn't available at the moment because it's not complete. I am creating this topic to make it like a personal journal in which I discuss the various concepts that I'm working on for this plugin. Also to keep me motivated and probably collect some thoughts from you (the script-writers) in the process of creating it.

Right now this plugin is just a bunch of scattered files used in my tests. I will start posting here the various concepts I plan to implement in this plugin. If the staff doesn't allow this kind of threads than please lock it. Please do not ask for a release date because it's just a work in progress (like the title says). Thank your for your time.

Also, please do not post something unnecessary here like "Thanks!", "I'll be waiting for this!", "You suck!" etc. Only your thoughts on the concepts that I discuss or some ideas you think will be useful.

Nice idea. Irrelevant posts will be removed to keep this topic as clean as possible.

Sweet, sweet, events. The beating heart of every system out there. They're the ones that tell you when to react and how you should react. To differentiate this plugin from the rest of them I had to do so major work here. So let's go slowly through the new event system in this plugin by analyzing the one in the current Squirrel plugin.

---

You all know the current event system and how it works. The Squirrel plugin has some various fixed function names that it expects in the global namespace of your script. Names like 'onServerStart', 'onServerStop', 'onPlayerJoin', 'onPlayerPart' etc.

function onServerStart()
{
print("We're live!");
}
function onServerStop()
{
print("We're off!");
}

function onPlayerJoin(i_player)
{
print("Welcome to our city!");
}

function onPlayerPart(i_player, reason)
{
print("Have a nice day!");
}
On each event the Squirrel plugin looks into the global table for an element with that name. If an element is found and that element is a function then the plugin executes that function and forwards the arguments received from the server. It's a nice and simple implementation but sadly not my style. It can get quite tedious when you start to rule out various calls based on the passed arguments. So let's move to the new event system and see the differences.

---

The new event system ditches global functions in favor of OOP (OOP is used internally, you don't deal with it). Since Squirrel functions can be treated like lambdas. That means you can capture a function in a variable if you want:

local a = function(a, b, c) { return a + b - c; }
 print( a(8, 3, 2) ); // Should output 9

Although lambdas in Squirrel look different and they're prefixed with '@' so they would look like ''local a = @(a, b, c) return a + b - c;". But both expressions work and that's not the point of this discussion.

The point was to see how a simple event would be declared in the event system I'm working on. So let's look at a simple one:
local my_event_listener = server.event(EVENTID.PLAYER_DISCONNECT);

my_event_listener.on_trigger = function(player, reason)
{
print("I got bored!");
}
First we create an instance of the event listener. The event listener automatically starts listening to the event associated with the ID that you specified. In this case the player disconnect event. But that doesn't do anything because when the event is triggered there's no one to listen or react. You can also pass a bool after the ID to specify if you want the event to be suspended or not.  What I mean by that is if you pass 'true' then the event will not trigger until you unsuspend it later on.

That event instance has a couple of member variables which will discuss in later posts. Because the event system is quite complex and has many features, each feature will be discussed in a separate post. Right now we're interested in the 'on_trigger' member variable. This variable contains the function you wish to get called when the event is triggered.

When the event is triggered it looks to see if there's a function inside the 'on_trigger' member variable. And if there is, then that function is executed and the arguments from the server are forwarded automatically. I also don't have to search the global table every time an event is triggered. Which may save some performance for large game-mods with a crowded global table.

If you look at this right now you might think it's a waste of time. But in later posts I will make you think otherwise. This is just an introduction to give you an idea of it looks. I'll see you in the next post ;)

In this post we explore the 'idle' member function which expects an integer. This is the ability of the event system to stay idle for a specified period of time. That means the trigger function you specified won't get triggered until that time has passed. Even if the event is triggered by the server internally, it's simply ignored. The idle time is specified in milliseconds. And can be specified wherever you want in the code. Even in the function you specified to be triggered.

Let's see how we can make a small timer using this functionality:

local my_event_listener = server.event(EVENTID.FRAME);

my_event_listener.idle = 5000; // Initial: Sleep for 5 seconds

my_event_listener.on_trigger = function(delta)
{
print("5 seconds have passed!");

my_event_listener.idle = 5000; // Sleep for another 5 seconds
}
We start by listening to the frame event because this is called very frequently. Then we tell it to sleep for 5000 milliseconds which means 5 seconds. We could avoid the initial sleep and allow the function to be called right away. But I just added that as an example.

Then in the triggered function that we specified we tell the event to sleep for another 5000 milliseconds each time the function is called. This isn't a very effective timer but it's an example of what you can do with this functionality. The final plugin timer will use a similar approach but in a much more optimized way.

See you in the next post with the next feature :)

In this post we explore the 'stride' member function which also expects an integer. This is the ability of the event system to skip a certain amount of event calls before triggering the specified function. Maybe you don't want your function to get executed every time the event is triggered.

Say you just want to give 1000$ to the 5'th player that connects to your server. Wouldn't it be cool if the event system would do that for you? Instead of you keeping track of this in some god forsaken variable somewhere in your script.

Let's see how we would use the 'stride' member variable to give $1000 to every 5'th player that connects to your server:
local my_event_listener = server.event(EVENTID.PLAYER_CONNECT);

my_event_listener.stride = 5;

my_event_listener.on_trigger = function(player)
{
player.money = 1000; // You lucky bastard ;)
}
We start as usual by attaching to the event we want. In this case the player connect event. Then we set the stride to 5. This means that our trigger function won't get called until this event was triggered 5 times in the server. After the event was triggered 5 times (internaly), our function gets executed and we receive the arguments associated with this event type as usual. Then this process starts over and the event is skipped for another 5 times.

Associated with this functionality is the 'ignore' member variable. Which can be used anywhere in your code to query how many times there's left to skip that event. So let's go step by step in this. Let's say that 3 players have connected since that event was created. You don't know that but somewhere in your script you realize that you need to know that. Well, you can always ask that from your event instance using 'my_event_listener.ignore' which, if we follow this fictional story should return 2. Because our stride is 5 and 3 players have connected so far....

You can also change the remaining times to skip using the 'ignore' member variable. Because that's a getter/setter you can simply adjust that anywhere in your code.

Let's say that every time a player disconnects you want to increase the remaining amount of events to ignore. Let's see how we would do that:

local my_event_listener = server.event(EVENTID.PLAYER_CONNECT);

my_event_listener.stride = 5;

my_event_listener.on_trigger = function(player)
{
player.money = 1000; // You lucky bastard ;)
}


local another_event_listener = server.event(EVENTID.PLAYER_DISCONNECT);

another_event_listener.on_trigger = function(player, reason)
{
my_event_listener.ignore += 1;
}
Now, each time a player disconnects, another player must connect to actually receive the $1000 prize. The stride still remains 5. You just altered the current times the event has to skip in the current iteration. Hell, you can even make it bigger then the stride if you want.  Except when all those events are skipped the value goes back to 5 like the stride. You can make it 100 and then the event will be skipped 100 times before being reverted back to the initialized stride value. Which in this case was 5.

In the next post we use what we have so far to get an idea of how the new event system works so far. Then we proceed to the remaining features :)

Using only those two features we'll create a small script that prints every 60 seconds how many players have connected/visited since the server was started. This isn't the way to do this but let's see how fun is to combine these :)

Let's have the first half of the code first and then proceed to explaining it:
local my_event_listener = server.event(EVENTID.PLAYER_CONNECT);

// This will be defined as the maximum value an integer can hold
my_event_listener.stride = SQMOD.INT_MAX;

my_event_listener.on_trigger = function(player)
{
// Dummy event and we don't care about it
}

We start by creating an event listener instance which is triggered each time a player connects to the server. And then we set a stride to some ridiculous number. In this case we use the compile time value defined in "SQMOD.INT_MAX". That define will be available in the plugin and is not a part of Squirrel API. On 32 bit platforms that value can be in the ranges of 2,147,483,647. You can see that is a ridiculous value to have as a stride.

We don't need this event to actually execute any function. In fact we don't even have to assign a trigger function. We just need it to count how many times it was triggered internally in the server. In this case how many players have connected. So that we can use the 'ignore' member variable later on to extract the actual amount of visitors.

Now let's have the other half of code and proceed to explaining it as well:
local another_event_listener = server.event(EVENTID.FRAME);

another_event_listener.sleep = 60000; // Sleep for 60 seconds initially

another_event_listener.on_trigger = function(delta)
{
announce("Visitors on our server: " + (my_event_listener.stride - my_event_listener.ignore);

another_event_listener.sleep = 60000; // Sleep for another 60 seconds
}
In this half of code we create another event listener instance which listens to the frame event. You can see that we're using the same trick we did earlier to get a temporary timer working. We make this event sleep for 60 seconds initially and then another 60 seconds after each call. Yay, we have a 60 seconds timer.

In the function we specified to be triggered we make use of the 'ignore' member variable from our previous event listener. The one with the stride. And we simply get how many times there's left to ignore that event. Then we subtract that from the stride we set and we get how many times that event was triggered.

Now that we know how many times have connected since the server started we make an announcement and make the event sleep for another 60 seconds. How easy was that?

I'll see you in the next post with the next feature :)

Some existing events are called way too frequently (example: onPlayerMove) and hence are largely unused due to performance issues.
Will the 'stride'/'sleep' functionality help that cause?

Quote from: aXXo on Feb 27, 2015, 11:02 PMWill the 'stride'/'sleep' functionality help that cause?

Definitely. But be patient and wait for the rest of the features that I'm documenting as we speak.  I'm working on the subsets now but the next feature which will be entity filtering and that will definitely help with this issue as well.

In this post we explore the benefits of subsets and what they are. There are only two subsets available and you have to use them wisely to not exhaust them. Those can be controlled by the 'primary' and 'secondary' member variables which also work with integers. The subsets allow you to filter the server events to listen only for what you need. Basically it allows you to take the arguments of a server event that is of type integer and see whether you want your function to be executed or not.

Probably you are a bit confused right now. So let's take this slowly by walking you through how the events work on the server. There are events in the server that are actually a group of events. Like a set of events, hence the name sub-set. Take the 'onPlayerActionChange' event. That single server callback is a group/set for 12 events (actually 13 if you count the null event) which are:
0 - none
1 - normal
12 - aiming
16 - shooting
41 - jumping
42 - lying on ground
43 - getting up
44 - jumping from vehicle
50 - driving
54 - dying
55 - wasted
58 - entering vehicle
60 - exiting vehicle
Let's see how this looks without subsets first by trying to figure out when a player is jumping from a vehicle:
local my_event_listener = server.event(EVENTID.PLAYER_ACTION_CHANGE);

my_event_listener.on_trigger = function(player, previous_action, current_action)
{
if (current_action == 44) {
print("Suicidal maniac!");
}
}
We start as usual by creating an event instance which listens to the player action change event from the server and then assign a function that we want to get triggered when that happens. Then in our assigned function we test if the current player action is the one that we need and print a simple message. Not bad, right? But I think we can improve that.

These events are are passed as an integer argument to your specified function. And then you have to do a bunch of unnecessary tests on those arguments if you only need one of them. Wouldn't be nice if the event system would do that for you? Would definitely save a lot of time and make things much simpler.

This is what subsets were designed for. They allow you to extract the parts you need from a server callback and completely ignore the rest of the calls. This takes some load of the script (and script-writer) and make things much more clear.

---

Remember when I told you that these subsets can be exhausted? Well, if you look at the event in the example above you can see that the event has two integer arguments. And how many subsets do we have? Exactly 2. Which means that only one subset per argument can be set.

In this case the primary subset will filter the previous player action and the secondary subset will filter the current player action for this server event. But what about a server event that only has one integer argument. Like the player disconnect for example which has the 'reason' argument. Does that mean we have one subset remaining which does nothing?

Not actually. In those callbacks both subsets are used for that argument. Which means that you can test for two things at a time. That's what I meant earlier when I said that there are only two subsets and it's important that I use them properly.

---

So, how do we make the above example to use these subsets that we talked about. Well, let's see:
local my_event_listener = server.event(EVENTID.PLAYER_ACTION_CHANGE);

// 44 is the id of the "jump from vehicle" action
my_event_listener.secondary = 44;

my_event_listener.on_trigger = function(player, previous_action, current_action)
{
print("Suicidal maniac!");
}
Again, we start as usual by creating our event. Except this time we make use of the 'secondary' member variable which controls the secondary subset. And if you remember correctly when I told you that because this server event has two integer arguments the secondary subset would be occupied by the second integer argument. And that argument is none other than the current player action.

Now, each time that event is triggered in the server the event instance that we created checks to see if the value in that argument matches the one in our subset. And if the values match then the event instance triggers the function that we assigned to it.

---

Now you might say: But you haven't used the primary subset. Well, in this case it wouldn't be that necessary but let's do that for the same server event. Let's make this event even more strict and tell it to only execute our function when the current action is 44 (jumping from vehicle) and the previous action is 50 (driving). Doesn't make too much sense but let's try that.

local my_event_listener = server.event(EVENTID.PLAYER_ACTION_CHANGE);

// 50 is the id of the "driving a vehicle" action
my_event_listener.primary = 50;

// 44 is the id of the "jump from vehicle" action
my_event_listener.secondary = 44;

my_event_listener.on_trigger = function(player, previous_action, current_action)
{
print("Suicidal maniac!");
}
This time our assigned function will only get called if the id of the previous action is 50 and the id of the current action is 44. That means it get's called only if the >driver< jumps from vehicle.

---

But how does an event with only one integer argument look like? (you might ask) Well, let's do that with the player disconnect event which has an integer augment.

local my_event_listener = server.event(EVENTID.PLAYER_DISCONNECT);

my_event_listener.primary = EPARTREASON.TIMEOUT;
my_event_listener.secondary = EPARTREASON.CRASHED;

my_event_listener.on_trigger = function(player, reason)
{
print("Something went wrong with this client!");
}
We start as usual by listening to the player disconnect event. Only this time we activate both subsets to make our function only get called when the player disconnected abruptly. Since there's only one integer argument then both subsets can be used on that single argument. Which means that the specified function only get's executed if the event argument matches either one of the subsets values we specified.

Also note that we used two built in compile-time values that will be available in the plugin (they're actually enumerations). These are 'EPARTREASON.TIMEOUT' which will be equal to the timeout reason ID (aka. 0) and 'EPARTREASON.CRASHED' which will be equal to the crashed reason ID (aka. 3). This is to allow the script to still function if those values are ever changed in the server. You should avoid hard-coded values whenever possible.

---

Subsets can be changed anytime and wherever you want in your code. This gives you more control over your script at run-time. They can also be deactivated by setting them to -1. But if you remember correctly I told you to avoid using hard-coded values. That value will probably be available through compile time define like 'SQMOD.DISABLE_SUBSET'. You probably also realized that subsets only work with positive values.

Now that we discussed about the subsets it's time to move to the next feature. So I'll see you in the next post :)

Time to discuss about entity filtering and how can we get events only from the entities we want and ignore the rest. This subject is pretty much self explanatory. The feature is still just on paper but it's too straight forward to not work. The only thing thing that might differ in the final version might be the member function names depending on how successful I get with the function overloading in Squirrel.

I haven't tested this yet but I think the function overloading in Sqrat is done by the amount of arguments instead of the argument types. Which means that some functions might have the entity names prefixed to them like 'filter_player()' instead of just 'filter()' where the argument type decides what to filter. Anyway, for the purpose of this post I'll assume that the function overloading will work based on data type and use non-prefixed names.

---

As usual we start with a common situation that demonstrates how this works. Time to put our imagination to work again. Let's say that you want to intercept every chat message of the spectating players. Let's start with the first half by creating an event that does that:
local my_event_listener = server.event(EVENTID.PLAYER_CHAT, true);

my_event_listener.on_trigger = function(player, message)
{
print("Spectator just said" + message);
}
We create the event listener instance to get the event we want and we assign the function to be triggered as usual. Except there's something different when we create the event instance this time. Did you noticed that 'true' argument after the event ID? If you remember correctly I told you that we can control whether the event instance should forward the server events immediately or be suspended until we enable it back.

If we wouldn't suspend this event instance then we would receive the chat messages from all the players (spectators or not). Because there's no player in it's entity filter at the moment. So, this time we make use of the 'suspend' member variable to control when this event should execute or not.

That event alone doesn't do anything so it's time to see the second half which controls that:
local another_event_listener = server.event(EVENTID.PLAYER_SPECTATE);

another_event_listener.on_trigger = function(player, target)
{
my_event_listener.grab(player);

if (my_event_listener.suspend) {
my_event_listener.suspend = false;
}
}

local yet_another_event_listener = server.event(EVENTID.PLAYER_SPAWN);

yet_another_event_listener.on_trigger = function(player)
{
my_event_listener.drop(player);

if (my_event_listener.player_count == 0) {
my_event_listener.suspend = true;
}
}
Like we did earlier we create the event we need and assign our function. Except this time there's two events that help us identify spectating players. So let's go through that code and see what it does. But first let's explain what those new member functions do since we haven't mentioned them before.

Like we discussed previously there might be a chance that you'll have these functions prefixed with an entity name but we'll ignore that for now. The 'grab()' member function takes the entity instance that you specify and adds it into the internal entity filter. The 'drop()' member function does the opposite by taking the entity instance that you specify and removing it from the internal entity filter.

So using these two functions we control which players we should trigger the event in the first half of code or not. In our first event we listen for the player spectate event. And for every player that triggers that event we add it to the filter of our first event instance.

But right now that event might be suspended (actually it is suspended, initially). So we check to see if it is suspended and unsuspend it using the 'suspend' member variable. When you make the 'suspend' member variable to 'true' then the event is suspended and every event call is ignored during that time. When that variable is set to 'false' then the event is no longer suspended and your function is executed again.

Using the second event that we created and which listens to the player spawn event. We control which players are no longer spectators. And we use the 'drop()' member function to remove them from the internal filter of the first event that we created.

In the last event function we also check to see there's any player entity in the filter. And if there's no entity in that filter then we get executed for each player chat message like in the beginning. So we need to suspend the event back when that happens.

In the final implementation you might not need to worry about suspending the event. Because there might be two event types Local and Global. This is because I have to reduce the amount code which manages the event distribution internally in the plugin. Which means that this feature is subject to change and might different in the final implementation.

---

There's a lot more member functions/variables involved with filters and they'll be discussed when the plugin is actually released. One of them being the 'inverse_*' member variables (prefixed for each entity type) which can reverse the effects of a filter. When you set them to 'true' the filter is inversed and when you set them to 'false' the filter works normally.

That means instead of receiving events from the entities in the filter you end up receiving events from entities which are not in the filter. This could get handy sometimes but it's also subject to change depending on how much code is required to manage the event distribution.

There's more to entity filtering but this is the basic approach that I'm taking at the moment. That's about it for tonight. I'll see you in the next post discussing other features I'm working on :)

https://bitbucket.org/stormeus/0.4-squirrel/commits/branch/rewrite-2.0

Quote from: ysc3839 on Feb 28, 2015, 04:02 AMhttps://bitbucket.org/stormeus/0.4-squirrel/commits/branch/rewrite-2.0

I'm quite aware of that.

Turns out I was right about method overloading in Squirrel. Actually I was expecting that :D Therefore the methods will be prefixed with the entity type name in the final version of this plugin. Also instead of having those grab_*()/drop_*() member methods which don't really express the intentions. I decided to rename them to include_*()/exclude_*() which means include this into the internal filter and exclude this from the internal filter.

Also after further investigations there will be 2 separate event types. Those are Global and Local. The global event will bind (internally) to the global event pools and listen for each call of the specified event. The local event on the other hand will only bind (internally) to the local event pools of each entity. Which reduces the overhead of calling functions who don't really need to be called.

Last time I also said there will be an option to inverse the internal filters. The previously presented implementation was also abandoned. The functionality still exists except in a different form. By default, the global event listener is has it's internal filter inversed. Which means that whatever is included in those filters will automatically stop at the C++ level and never reach the script level to trigger the specified function.

The local event listener has a normal filter by default. Not because the global event filter is inversed by default but only because it's the way is supposed to be. If you remember correctly I said that the local filter will only bind to the local event pools for each specified entity. And therefore its only normal for this event type to only listen to the entities in the internal filters.

There will also be a few other binding points that you can use in the event instances and not just the usual 'on_trigger'. Next to that there will also be:

• on_included - Called each time an entity is added to the internal filters
• on_excluded - Called each time an entity is removed from the internal filters
• on_cleared - Called each time an internal entity filter is cleared/emptied/reset.
• on_destroyed - Called each time and entity is destroyed elsewhere and must be removed from the filters.

The above binding points will also reduce the amount of code required to react/addapt to filter changes in other places where entities are added or simply removed from the filters.

There might be a few other changes to the entity filtering system but for now that's the general idea. The current changes in the event system are meant to reduce the amount of C++ code required to manage the event distribution internally and to have a clean implementation.  I won't hurry the plugin release because I want to have a strong implementation that will last for a long time :)

Project was postponed after a rage quit this night. The implementation was close to completion but that f*ing compiler failed to do as told and as soon as I started to add more code to the implementation I'd get:
Plugin error >> LoadLibrary() 'plugins/sqmod.dll' failed: Code 998Failed to load plugin: sqmod
That error code is a b!tch to debug. It's triggered by the smallest things. For example the first time I got that message it was because of a `Sqrat::RootTable` as a member inside a class. And as soon as I changed that to a pointer it began to work again. Until the next thing happened and it started again and again until I just said "f* you!" and quit.

So after I took the pros and cons of the project I realized it's just not worth the trouble.

• Too much crap to deal with.
• Some limitations in the server just make my life a nightmare.
• I could use the time for something better.

Anyway, I might continue to work on it another time, perhaps when the server improves a little and I find that it's worth the time and trouble. I'll refrain from writing more right now because I'm quite pissed of and bad things come out when I do that.

After taking some time off and listening to some music to take the steam off and relax a little. I decided to give one more try and see why that god damn error pops out of nowhere. Previous searches on google gave me sh!t, literally sh!t. Just some whiny people on some forums with similar problems for 3DS Max, DirectX, Cinema something.. etc. And the common answer was, "Upgrade your software." and then miraculously "The bug was fixed.". And I was like. Well... how the f* does that help me? Obviously the "upgrade your software" wouldn't apply to me.

So I decided to go deeper with the searching and luckily I found a few crumbs of information somewhere on some forsaken topics. Which suggested that I should pay attention to some of my static variables. And so I did. Ending up starting the project again, from scratch and trying to eliminate static variables as much as possible. Turns out this was it. The nightmare that kept me so frustrated these days was finally gone. I still don't know the exact cause and which static variable was the culprit. I'm just relieved that it's gone.

So I'm happy to announce that the project development was resumed. With a slower pace now. Mainly because I lost my interest after wasting a lot of time and on that error. At least now I know that I can safely write code that I don't have to throw away later on. That error must have been the cause of almost 5 re-writes of the plugin. I just couldn't understand what was going on.

Finally, I got a basic/untested implementation of the GlobalEvent system. Now comes the hard part. Implementing the LocalEvent system. And starting to test what I've implemented so far. The LocalEvent system is going to be harder to tame because it has to be extremely efficient and it involves a ton of internal management for which I have to find a clever way of keeping it to a minimum :P

---

---

I still haven't implemented the structures for entities and other things because from what I've noticed in Stormeus's version of the Squirrel plugin there will be some changes in the API so I'll have to wait until the changelog is released before I begin working on those. Right now I have just the basic structures for them to allow me to work on the rest of the plugin. More updates coming soon ;)

---

Tons of new features are planned:

• Local/Global user data and tags for custom personalization. Status: Almost completed but tested only quickly (http://postimg.org/image/ibp77s4b5/).
• Entity Queues for distribution of entity changes. Status: Incomplete. Awaiting until the changelog is released.
• Area System for custom collisions. Status: Half complete. Awaiting for an alpha/preview release first.
• Extensive built-in library. Status: Incomplete. Awaiting for an alpha/preview release first.
• etc.

These will be discussed here when I get the chance to implement them.

I got 99 problems but an event system ain't one of them. :P Just finished implementing a primitive version of the event system. Took a bit longer than I expected because the VC:MP server is a b!tch sometimes and debugging it is painful. But that's not gonna stop me :D After having a few more wasted prototypes I ended up having 3 event types instead of 2. I had to make another event type which doesn't include entity filtering when it's not needed. Mostly for performance reasons. So now you have BasicEvent, GlobalEvent and LocalEvent at your disposal.

---

---

Now that I got this out of the way I can start going through the rest of the source to do a few more improvements and probably look for some bugs. And later tonight, upload the source for anyone who wants to see what's new and what I'm working on. The implementation will be incomplete of course but should be functional for testing and learning about the plugin.

Did a small and very unrealistic benchmark on the current implementation to see how many events I can attach on the onFrame callback to get an idea of how it performs. So I started by creating one huge file with about 250 events that attach to the on frame event. Something like this:
local event1 = BasicEvent(EVENTID.SERVER_FRAME);

event1.on_trigger = function(delta)
{
    ++_I;
}

local event2 = BasicEvent(EVENTID.SERVER_FRAME);

event2.on_trigger = function(delta)
{
    ++_I;
}

local event3 = BasicEvent(EVENTID.SERVER_FRAME);

event3.on_trigger = function(delta)
{
    ++_I;
}

.....
..........
...............


local event250 = BasicEvent(EVENTID.SERVER_FRAME);

event250.on_trigger = function(delta)
{
    ++_I;
}

Then I duplicated that file about 50 times. At the end I created one more file with one event to count how many events were called in that frame. Something like this:
_I <- 0;

local event = BasicEvent(EVENTID.SERVER_FRAME);

event.on_trigger = function(delta)
{
    print(delta + " ---- " + _I);
    _I = 0;
}

Then I added those files to the configuration file to be loaded on start-up:
[Scripts]
Source = bench/1.nut
Source = bench/2.nut
Source = bench/3.nut

.....
..........
...............

Source = bench/50.nut

Source = bench/X.nut
The X.nut file is the last binded event and it's the one counting the calls. Which means that we should get about 12,501 events binded to the onFrame callback. Here's a screen:

The first number is the time since last frame. I'm assuming it's milliseconds because I multiplied that float number by 1000 to get an integer. The second number is the number of events triggered before it reached the last event which is in the file X.nut. The reason I said that this is an unrealistic benchmark is that in a real situation an actual script wouldn't use such simple functions. But still 12,501 events executed in just 27 milliseconds shouldn't be that bad (if that number is indeed milliseconds).

So that's about it. A small idea of how the event system performs :)

The repository (https://github.com/iSLC/VCMP-SqMod) was made public. I'll try to upload the source before I go to sleep and if possible, also leave a x32 bit dll available for download just for anyone who wants to test anything. I'll also begin working on the wiki (which can be found on the repository) to have a lightweight documentation available for anyone who can't get the information directly from C++.

Like I said, it's incomplete and probably looks like a primitive implementation, but it's a start. I'll slowly complete the source and wiki in the days to come.

Edit:

Temporary download link for x32bit binaries with a hello world script can be found here (http://expirebox.com/download/57d5efa889a7fcb873a16a7b4ef569a9.html).

Latest updates to the repository include:

• Entity iterators implemented. (Untested)
• Entity data types implementation completed. (Untested)
• Entity queues partially implemented. (Untested)
• Event filter iterators. (Untested)
• Event filter hint/specifier. (Untested)
• Partial miscellaneous data types implementation. (Untested)
• Various things removed until further testing.
• Tons of other changes and fixes too minor to mention.

Temporary download link for x32bit binaries with a hello world script can be found here (http://expirebox.com/download/3cdbce44de5af7bb7cfa9de3992c9c83.html).

Lately I've been thinking how to have a better administration system that everyone can use. So, I've spent some time thinking about it. But nothing ideal came to my mind for a while. Until I've remembered that I can register custom functions in SQLite. So, I've been thinking of a small hack (so to speak) to use SQLite to my advantage.

At run-time I would keep a database in :memory: that would track everything that's happening in the server. And then I'd register a bunch of functions to manipulate that data. So, now I have an expressive management system that I can use. Imagine if you could simply execute strings like:

select player_ban(id) from [players] where country_code = 'US';Bans all players from US

select player_give_money(id, 300) from [players] where class = 1;Gives 300$ to all players with the class id 1

select player_ban(id) from [players] where name = 'MasterX';Bans the player with the name 'MasterX'

select player_pos_near(id, 143.34, 764.2342, 565.643)
    from [players] where class = 0 limit 3;Teleport 3 players with the class id 0 near the specified position

select player_health(id, 0) from [players] where vehicle = 4 offset 2;Kill players in the vehicle with id 4 and start from the third player

There's way more things you can do with this. That's just a tiny example. Imagine how much code you'd have to write to do that and now you just have to execute a query string to do it.

I have a small example of this implemented in C++ to prove the concept here (http://pastebin.com/87Kq4zt4). Anyway, this is something I've been thinking about and I'm curious how will it perform. Would you find this useful? I wonder if it's worth to implement something like this and whether it's safe or not.

I forgot to mention guys but the development of this plugin was resumed here (https://github.com/iSLC/VCMP-SqMod). It's a completely revised version of the previous one that had some limitations. It's still under development and when I get the time I'll explain the new features that I've added and the issues that I've come across which forced me to revise the whole thing.

EDIT: If you have any suggestions, requests, bugs to report then you can always find me at @LUNet (http://liberty-unleashed.co.uk/irc/servers/) on channel #SLC (https://search.mibbit.com/networks/LUnet/%23SLC)

Keep in mind that the plugin is barely 15% completed and some of the features that I have in plan are missing currently.

Added hashing to the standard library:
local crc32 = CHashCRC32(); // Create CRC32 encoder
local keccak = CHashKeccak(); // Create Keccak encoder
local md5 = CHashMD5(); // Create MD5 encoder
local sha1 = CHashSHA1(); // Create SHA1 encoder
local sha256 = CHashSHA256(); // Create SHA256 encoder
local sha3 = CHashSHA3(); // Create SHA3 encoder

local data = @"f*in awesome";

local crc32_hash = crc32.compute(data);
local keccak_hash = keccak.compute(data);
local md5_hash = md5.compute(data);
local sha1_hash = sha1.compute(data);
local sha256_hash = sha256.compute(data);
local sha3_hash = sha3.compute(data);

if (HashCRC32(data) == crc32_hash) print(crc32_hash);
else print("individual hash didn't match with encoder instance");

if (HashKeccak(data) == keccak_hash) print(keccak_hash);
else print("individual hash didn't match with encoder instance");

if (HashMD5(data) == md5_hash) print(md5_hash);
else print("individual hash didn't match with encoder instance");

if (HashSHA1(data) == sha1_hash) print(sha1_hash);
else print("individual hash didn't match with encoder instance");

if (HashSHA256(data) == sha256_hash) print(sha256_hash);
else print("individual hash didn't match with encoder instance");

if (HashSHA3(data) == sha3_hash) print(sha3_hash);
else print("individual hash didn't match with encoder instance");

Working on an adding a fully featured IRC client on the standard library as well. It's actually implemented. I just need to work on the event processing loop to integrate it into the server frame event. Next post will be about the integrated IRC client.

The IRC implementation is almost done. All that's left to implement is the DCC and file transfer things. Which is not something that even makes sense to have on a game server. But because this is supposed to be a fully compliant IRC client, why not.

A sample of how to it looks like:
// A single IRC session can connect to one IRC server but multiple channels on that server
local irc_session = IRC.Session();

// Create a basic event that listens to the server start up event
local startup_event = BasicEvent(EVENTID.SERVERSTARTUP)
// Tell the event instance to call this function when it's being triggered
startup_event.on_trigger = function()
{
    // Connect the IRC session to this an IRC server (on success it returns 0)
    // Args: server_name, port, nickname (optional: password, username, realname)
    if (irc_session.connect("irc.liberty-unleashed.co.uk", 6667, "SqMod") != 0)
    {
        // Output the last error message (if any)
        print("Unable to connect to IRC server: " + irc_session.err_str);
    }
    // Can't join channels here because the session didn't connect yet

    // Tell the session to remove extract SLC from [email protected]
    // automatically when sending the `origin` to events
    irc_session.set_option(EIRCOPT.STRIPNICKS);
}

// Create a basic event that listens to the server shut down event
local shutdown_event = BasicEvent(EVENTID.SERVERSHUTDOWN)
// Tell the event instance to call this function when it's being triggered
shutdown_event.on_trigger = function()
{
    /* The session is closed automatically so... */
}

// Tell the irc session instance to call this funtion when it successfully connects to the server
irc_session.on_connect = function(event, origin, params)
{
    // Confirm in console
    print("Connected to IRC...");
    // Join a channel (on success it returns 0)
    if (irc_session.cmd_join("#SLC") != 0)
    {
        print("Unable to join channel: " + irc_session.err_str);
    }
    // Say something on some channel or user
    else
    {
        // Say a message in #SLC channel
        irc_session.cmd_msg("#SLC", "Hi. I am SqMod");
        // Say a message to SLC user
        irc_session.cmd_msg("SLC", "Sup?");
        // Do a /me command in #SLC channel
        irc_session.cmd_me("#SLC", "goes home");
    }
}

// Tell the irc session instance to call this funtion when someone says something in the channel
irc_session.on_channel = function(event, origin, params)
{
    // event is the event type in IRC [CHANNEL]
    // origin is who said the message
    // params[0] is the channel where the message was said
    // params[1] is the text that was said

    // See if the used who said this message was SLC
    print( format("%s said on %s : %s", params[0], origin, params[1]) );

    // Because the EIRCOPT.STRIPNICKS was enabled
    // `origin` will contain the clean user name instead of [email protected]
}

// Using IRC to evaluate code on the server from private messages

// Tell the irc session instance to call this funtion when someone when someone sent you a message
irc_session.on_priv_msg = function(event, origin, params)
{
    // event is the event type in IRC [CHANNEL]
    // origin is who said the message
    // params[0] is the channel where the message was said
    // params[1] is the text that was said

    // See if the used who said this message was SLC
    if (origin == "SLC")
    {
        // Compile message as valid script code
        local code = compilestring(params[1]);
        // Execute it
        code();
    }

    // Because the EIRCOPT.STRIPNICKS was enabled
    // `origin` will contain the clean user name instead of [email protected]
}

How to get a list of users on channels:

// Table of channels and arrays of connected users
IRC_USERS <- { /* ... */ };

// Create a basic event that listens to the server frame event
local fake_timer = BasicEvent(EVENTID.SERVERFRAME);
// Tell the event instance to sleep for 3000 milliseconds (3 seconds)
fake_timer.idle = 3000;
// Tell the event instance to call this function when it's being triggered
fake_timer.on_trigger = function(delta)
{
    // Ask the server for a list of users on #SLC channel
    irc_session.cmd_list("#SLC");

    print("------ List of users retrieved every 3 seconds");
    // Print the list of users retrieved on the last call
    foreach (chanel, users in IRC_USERS)
    {
        print("Users in " + chanel);
        foreach (user in users)
        {
            print(user);
        }
    }

    if (IRC_USERS.rawin("#SLC") && IRC_USERS.rawget("#SLC").find("[member=135]Doom_Kill3R[/member]") != null)
    {
        print("Found the nab");
    }

    // Tell it to sleep for another 3000 milliseconds (3 seconds)
    fake_timer.idle = 3000;
}

// Tell the irc session instance to call this funtion when an IRC RFC code is sent
irc_session.on_numeric = function(event, origin, params)
{
    switch (event)
    {
        case 353: /* RFC 1459 RPL_NAMREPLY */
            // params[0] Our nick name
            // params[1] = ? (dunno what this is)
            // params[2] Channel
            // params[3] List of users separated by space

            IRC_USERS.rawset(params[2], split(params[3], " "));
        break;

        case 366: /* RFC 1459 RPL_ENDOFNAMES */
            // params[0] Our nick name
            // params[1] Channel
            // params[2] End of /NAMES list.
        break;

        default:
            /* Unknown or unhandled RFC code */
    }
}


I've been working on a simple builtin command system to avoid having to see people writing a god function containing all commands and to take out the duplicate code used to extract command arguments and validate types.

The command system is very simple and works similarly to the event system. You begin by creating a command listener and specify a command name and optionally some argument type filters.

Just to be clear by invoker I mean the player who called the command.

So let's begin simply by creating a command listener that listens to the "test" command:
local cmd_test = Cmd.Listener("test");
Now in order to actually do something when this command is used we bind a function to the `on_exec` slot:
cmd_test.on_exec = function(invoker, args)
{
    print("test cmd called by " + invoker);
}
The function will receive the ID of the player that invoked the command and an aray with the passed arguments. A command supports up to 12 arguments to be specified if necessary.

So let's print the type and value of each received argument by modifying that function a bit:
cmd_test.on_exec = function(invoker, args)
{
    print("test cmd called by " + invoker);

    foreach (i, v in args)
    {
        print("type <" + typeof(v) + "> arg " + i + " has value: " + v);
    }
}
Now let's assume that someone calls this command using the following:
/test some_text "text with spaces" 786 false 'some other text' 2.343
The out might be as follows:
test cmd called by 0
type <string> arg 0 has value: some_text
type <string> arg 1 has value: text with spaces
type <integer> arg 0 has value: 786
type <bool> arg 0 has value: false
type <string> arg 0 has value: some other text
type <float> arg 0 has value: 2.343

As you can see the internal parser splits arguments where a space is found. However you can include a text as a single argument by enclosing it in single or double quotes. Similar to what Squirrel does. If the value is not enclosed in quotes then it tries to convert it to an integer and if that fails then to a float and if that fails then to a boolean and finally falling back to a string.

As its the case for the `some_text` argument which is not enclosed in quotes but it's a string. Obviously that cannot convert to an integer, float or boolean so it's used as a string. Argument two on the other hand `text with spaces` instead of being considered three distinct arguments it'll be grouped into a single one because of the quoting.

Including quotes inside quotes works the same way as in squirrel. Simply by escaping them with the back slash character \' and \" respectively.

But that's not the thing that saves up most of the code. The thing that prevents you from writing a ton of bloat code is the integrated type checking. You can specify what kind of types of values you command may receive as arguments.

Let's begin by adjusting the creation to also include type checking:
local cmd_test = Cmd.Listener("test", "i,f|s|b");
Hopefully you notified the second constructor argument value "i,f|s|b". That's a string which specifies what type of value each argument expects. The pipe '|' character separates the arguments while the comma separates the allowed types which are:

• i for integer
• f for float numbers
• b for booleans
• s for strings

Basically the second string specified previously tells that the command may only receive integer or float in the first argument, string as the second argument and boolean as the third.

Therefore if we try to call the command with something like this:
/test turbo "goes here" bool
Will throw an error saying that argument 0 (yes, argument counting starts from 0) is incompatible. Since it expects an integer or float and we passed the string `turbo`.

If you wish to catch that message and send information to the player then you can set a custom error handler:
Cmd.set_on_error(function(type, msg) {
    switch (type)
    {
        case ECMDERR.UNKNOWN_COMMAND:
            // Get the player that invoked the command and send him a message
            Cmd.get_invoker().message("No such command exists");
        break;
        case ECMDERR.UNSUPPORTED_ARG:
            // Get the player that invoked the command and send him a message
            Cmd.get_invoker().message("Wrong command arguments");
        break;
        case ECMDERR.EXECUTION_FAILED:
            // Get the player that invoked the command and send him a message
            Cmd.get_invoker().message("Failed to execute the command");
        break;
    }
});
Allowing you to customize what you want to say and how do you want to react when something fails.

But what if I don't want anyone to use my command? Well, there's something for that as well. There's another slot besides `on_exec` caled `on_auth` which takes a function that must return true or false if the invoker is allowed to use the command. But before that the command must have explicit authority turned on. That's done through the `auth` member. Let's see how that looks:

cmd_test.auth = true;
cmd_test.level = 3;

cmd_test.on_auth = function(invoker)
{
    local player = CPlayer(invoker);
    if (player.level >= cmd_test.level)
    {
        return true;
    }
    return false;
}
So let's see what we did. We enabled explicit authority clearance by setting `auth` property to true. We used the built in property `level` to store a number and use that as a way of saying that our command needs a level higher or equal to three to be called.

The same `level` property is built into players as well. This is provided by the plugin and it's optional if you don't have more complex systems. Each player can have it's own level. The level is nothing but an integer. Preferably the integer should start from 0. And let's say that a basic user has a level of 1 a vip user has a level of 2 a moderator has a level of 3 and an admin has a level of 4.

Basically we said that our command needs to be called by someone who's at least a moderator to be allowed to call it. Finally we set out custom function which inspects the invoker level and compares it with our command to see if the execution should be allowed.

Of course you don't have to use the built in authority system and you may implement your own with more fine grained privileges. For example, with the built in level system we don't even need to set a function to the `on_auth` slot.

We just have to enable explicit authority by setting `auth` to true and set the minimum level required to executed the command. if the level is not less than 0 (negative) then it will be used to compare it to the invoker's level.

There are tons of other customization. Setting a minimum and maximum allowed arguments required to call your command. Suspend command listener to prevent further calls. Set up a help or informational text or generate them automatically based on the argument types specified and other things.

Help describes what the command is supposed to do and what it needs. While information describes how to be used. Basically that means the syntax that is allowed.

Dynamically update it's name and specifiers. And a ton of other things to come. This was meant to prevent you from having to write a ton of code for checking types and dealing with permissions. It's still a work in progress but it's something and worked just fine in the basic tests.

UPDATES (major only):

• Dropped the current design and every relation to C++11. Apparently Squirrel and Sqrat (the binding utility) don't behave well in C++11. So now the project is back on C++03
• Implemented the INI library. (examples in following posts)
• Implemented the XML library. (examples in following posts)
• Various other libraries partially implemented. Such as Time, Random, Numeric etc.
• Dropped the previous event system due to its complexity in favor of a more simplistic one allowing and recommending OOP designs.

Examples of new mentioned features will be available in the following posts.

Implemented the INI library featuring Unicode support (which is irrelevant since the plugin uses ASCII), As well as MultiKey and MultiLine support. As implied, the MultiKey feature allows INI files to have multiple keys with the same name under a section. While MultiLine allows values to span across multiple lines.

For the sake of this example, I'll use a string as the source instead of a file. Also, the things demonstrated here only show a portion of what can be achieved with the current implementation.

Assuming we have the following INI:
local doc = @"
[Database]
Host=127.0.0.1
Port=5443

[Options]
Test=23
Test=false
Test=Text goes here

[Data]
Text= <<<MULTI_LINE
Multi line
    content
allowed
MULTI_LINE
";
We begin by creating an INI document. And load the string into the instance. If any errors occur, a proper exception is thrown.
local ini = SqIni.Document(false, true, true);

ini.LoadData(doc);

The three arguments passed when creating the document disable UTF8 and enable MultiKey and MultiLine.

So now that we parsed the string and we have data in our document. Let's try reading individual values:
print(ini.GetValue("Database", "Host", "*"));
print(ini.GetInteger("Database", "Port", 3432));
print(ini.GetValue("Database", "Name", "default value"));
The first two arguments describe the section and the key from which the value should be retrieved. The third one is a value that should be returned if that key doesn't exist. And if everything went well, we should get the following output:

[USR] 127.0.0.1
[USR] 5443
[USR] default value
How about getting all sections and printing them? Let's try:
local inisec = ini.GetSections();

printf("Sections: %d", inisec.Size);

while (inisec.Valid)
{
    print(inisec.Item);
    inisec.Next();
}

First we obtained a list of sections and then we tested to see if the returned iterator points to a valid section. If it does, then we print it's name and move to the next one. Then we test again if the iterator points to a valid section and print it again. And we do so until we reach the end.

But how about those multi keys under the [Options] section? Well, we apply almost the same method that we did with sections:

local inival = ini.GetValues("Options", "Test");

printf("Keys: %d", inival.Size);

while (inival.Valid)
{
    print(inival.Item);
    inival.Next();
}

First we retrieve a list of values from all the keys named "Test" under the "Options" section. And we iterate over them like we did with sections. Optionally, we can sort them into the order they were defined in the INI file if that's a requirement inival.SortByLoadOrder();

And sure enough, we should have the following output:
[USR] Keys: 3
[USR] 23
[USR] false
[USR] Text goes here
So now lets try the multi line text. Which we can access just like an individual item:
print(ini.GetValue("Data", "Text", ""));
Output should be:
[USR] Multi line
    content
allowed
This is only a small example. You can modify documents, save them and much more.

This is a demonstration of the built in time library that allows easy manipulation of time with microsecond precision. This is a small demonstration and more features remain to be implemented.

NOTE: SLongInt is part of the Numeric library and allows the user to work with 64 bit integers on 32 bit builds.

print( "[0]  " + time() ); // Only option in squirrel
// Options in this plugin
print( "[1]  " + SqTime.Now().SecondsI );

print( "[2]  " + SqTime.Now().Milliseconds );
print( "[3]  " + SqTime.Now().Milliseconds / SLongInt(1000) );
print( "[4]  " + SqTime.Now().Microseconds );
print( "[5]  " + SqTime.Now().MinutesI );
print( "[6]  " + SqTime.Now().HoursI );
print( "[7]  " + SqTime.Now().DaysI );
print( "[8]  " + SqTime.Now().YearsI );

print( "[9]  " + ( SqTime.Now() + SqTime.Seconds(10) ).SecondsI );

print( "[10]  " + SqTime.Days(1.5).HoursI );
print( "[11]  " + SqTime.Years(1).DaysI );
print( "[12]  " + SqTime.Minutes(5.2).SecondsI );
print( "[13]  " + SqTime.Minutes(45).HoursF );
print( "[14]  " + SqTime.Milliseconds(10000).SecondsI );

print( "[15]  " + ( SqTime.Minutes(2) - SqTime.Seconds(30) ).SecondsI );

NOTE: The suffixed I and F at the end stand for Integer and Float. Since sometimes you need to preserve precision with fractional numbers.

Output:
[USR] [0]  1456131000
[USR] [1]  1456131000
[USR] [2]  1456131000991
[USR] [3]  1456131000
[USR] [4]  1456131000991210
[USR] [5]  24268850
[USR] [6]  404480
[USR] [7]  16853
[USR] [8]  46
[USR] [9]  1456131010
[USR] [10]  36
[USR] [11]  365
[USR] [12]  311
[USR] [13]  0.75
[USR] [14]  10
[USR] [15]  90

---

local tm = SqTime.Timer();

// Performing some arbitrary computations
for (local i = 0; i < 10000000; i++)
{
    i += 1;
    i -= 1;
}

print("Microseconds elapsed: " + tm.Elapsed.Microseconds);
Output:
[USR] Microseconds elapsed: 525408

This in response to a popular debate here on the forums on who's faster INI or SQLite. The series of these benchmarks are meant to show that both are decent candidates for storing data if implemented well enough.

These benchmarks should not be taken too seriously as they rely on the fact that the user needs to process large amounts of data in bulk. Which in real code it's rarely the case.

---

Quote from: aXXo on Mar 22, 2016, 10:53 PM[Benchmark] SQLite vs INI (http://viceunderdogs.com/index.php?topic=2228.0)

Unfortunately that benchmark is a bit outdated and tbh it is unfair. None of the plugins/modules make use of all the features that SQLite provides or does any kind of optimizations as is the case of the INI plugin/module. Why? Because the entire INI file is parsed every time you need to read/write something and the SQLite benchmark does not make use of transactions and/or pre-compiled statements. And the simulation is not fair or process the same amount of data.

I am going to run a series of benchmarks and describe further methods of optimizations which I've implemented in my plugin during it's development. Hopefully by the end of this benchmark you will get and idea how further you can go with optimizations if the storage is the bottleneck.

---

I'll be using the following code to time my benchmarks and generate a nice and structured output:
local _Timer, _Count;

function StartBenchmark(name)
{
    print(SqStr.Center(name, '-', 70));
    print(""); // Just some spacing
    _Timer = SqTime.Timer();
    _Count = 0;
}

function StepBenchmark(step)
{
    local elapsed = _Timer.Restart().Microseconds.Num;
    SqLog.Inf("%s:%s microseconds",
                SqStr.Left(step+"", ' ', 16)
                SqStr.Right(elapsed+"", ' ', 12));
    _Count += elapsed;
}

function StopBenchmark(name)
{
    SqLog.Inf("%s:%s microseconds",
                SqStr.Left("Total", ' ', 16)
                SqStr.Right(_Count+"", ' ',  12));
    print(""); // Just some spacing
    print(SqStr.Center(name, '-', 70));
}


For this benchmark I will use a single INI file and write 10,000 dummy accounts:

StartBenchmark(" INI WRITE ");

local ini = SqIni.Document(false, false, false);

StepBenchmark("Creation");

for (local i = 0; i <= 10000; ++i)
{
    local section = "Account_" + i;
    ini.SetInteger(section, "ID", i);
    ini.SetValue(section, "Name", "user_" + i);
    ini.SetValue(section, "Password", "password_" + i);
    ini.SetBoolean(section, "Active", i % 2 == 0);
}

StepBenchmark("Insertion");

ini.SaveFile("Accounts.ini");

StepBenchmark("Saving");

StopBenchmark(" INI WRITE ");

Benchmark results:
[USR] ------------------------------ INI WRITE -----------------------------
[USR]
[INF] Creation        :          32 microseconds
[INF] Insertion       :      308865 microseconds
[INF] Saving          :       93330 microseconds
[INF] Total           :      402227 microseconds
[USR]
[USR] ------------------------------ INI WRITE -----------------------------
Contents of the new INI file:
[Account_0]
ID = 0
Name = user_0
Password = password_0
Active = true


[Account_1]
ID = 1
Name = user_1
Password = password_1
Active = false

...

[Account_10000]
ID = 10000
Name = user_10000
Password = password_10000
Active = true

Optimizations behind the scene:

• When inserting elements, they are created in memory. The only time disk IO is necessary is when writing everything at once.

Bottlenecks behind the scene:

• The internal containers have to scale on each insert and therefore some allocations have to occur. If this was a pressing matter I could've used some methods of pre-allocating space upfront. But unfortunately it's not.
• I'm suspecting that each element is written to the file individually rather than writing it all in a memory buffer and then to disk. I'll test this in the further optimization stage.

---

For this bench mark I'll be reading and processing the file I've previously created and also do some minor validation on the data:

StartBenchmark(" INI READ ");

local ini = SqIni.Document(false, false, false);

StepBenchmark("Creation");

ini.LoadFile("Accounts.ini");

StepBenchmark("Loading");

local acc_id, acc_name, acc_pass, acc_active;

for (local i = 0; i <= 10000; ++i)
{
    local section = "Account_" + i;

    local acc_id = ini.GetInteger(section, "ID", -1);

    if (acc_id != i)
    {
        throw "account has a bad ID";
    }

    acc_name = ini.GetValue(section, "Name", "");
    acc_pass = ini.GetValue(section, "Password", "");
    acc_active = ini.GetBoolean(section, "Active", false);
}

StepBenchmark("Processing");

StopBenchmark(" INI READ ");

Benchmark results:
[USR] ------------------------------ INI READ ------------------------------
[USR]
[INF] Creation        :          32 microseconds
[INF] Loading         :       82465 microseconds
[INF] Processing      :       90007 microseconds
[INF] Total           :      172504 microseconds
[USR]
[USR] ------------------------------ INI READ ------------------------------
Optimizations behind the scene:

• The entire file is read into memory at once and disk IO is no longer necessary after.

Bottlenecks behind the scene:

• The internal containers have to iterated whenever a value is retrieved. I'm planning to further optimize this later by allowing to fetch every key/value pair in a section at once.

For this benchmark I'll create a table [Accounts] and insert 10,000 dummy accounts in it:

StartBenchmark(" SQLITE WRITE ");

local db = SQLite.Connection("test.db");

StepBenchmark("Connection");

db.Exec(@"CREATE TABLE IF NOT EXISTS [Accounts] (
    [ID] INTEGER  PRIMARY KEY NOT NULL,
    [Name] VARCHAR(32)  UNIQUE NOT NULL,
    [Pass] VARCHAR(128)  UNIQUE NOT NULL,
    [Active] INTEGER DEFAULT 0 NULL
);");

StepBenchmark("Creation");

for (local i = 0; i <= 10000; ++i)
{
    db.ExecF("INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (%d, 'user_%d', 'password_%d', %d);"
                i, i, i, i % 2);
}

StepBenchmark("Insertion");

StopBenchmark(" SQLITE WRITE ");

Benchmark results:
[USR] ---------------------------- SQLITE WRITE ----------------------------
[USR]
[INF] Connection      :         407 microseconds
[INF] Creation        :      116224 microseconds
[INF] Insertion       :   742909447 microseconds
[INF] Total           :   743026078 microseconds
[USR]
[USR] ---------------------------- SQLITE WRITE ----------------------------
At this point you're probably like: WTF SLC? 12 Minutes! How can SQLite be thousands of times slower than INI files? And now you've let me know that you didn't pay attention to the benchmark title. Otherwise you'd have seen that uppercase/bold/red word.

Quote from: SQLite FAQ(19) INSERT is really slow - I can only do few dozen INSERTs per second

Actually, SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second. Transaction speed is limited by the rotational speed of your disk drive. A transaction normally requires two complete rotations of the disk platter, which on a 7200RPM disk drive limits you to about 60 transactions per second.
Transaction speed is limited by disk drive speed because (by default) SQLite actually waits until the data really is safely stored on the disk surface before the transaction is complete. That way, if you suddenly lose power or if your OS crashes, your data is still safe. For details, read about atomic commit in SQLite..

By default, each INSERT statement is its own transaction. But if you surround multiple INSERT statements with BEGIN...COMMIT then all the inserts are grouped into a single transaction. The time needed to commit the transaction is amortized over all the enclosed insert statements and so the time per insert statement is greatly reduced.

Another option is to run PRAGMA synchronous=OFF. This command will cause SQLite to not wait on data to reach the disk surface, which will make write operations appear to be much faster. But if you lose power in the middle of a transaction, your database file might go corrupt.

Source. (http://www.sqlite.org/faq.html#q19)

Optimizations behind the scene:

• N/A

Bottlenecks behind the scene:

• Because each query has it's own transaction and transactions are very slow. Inherently makes each individual query take a significant amount time.
• Each query has to be compiled individually. It's the same as compiling your script 10,000 times for no reason at all.
• Others which will be discussed later.

---

For this bench mark I'll be reading and processing the file I've previously created and also do some minor validation on the data:

StartBenchmark(" SQLITE READ ");

local db = SQLite.Connection("test.db");

StepBenchmark("Connection");

local acc_id, acc_name, acc_pass, acc_active;

for (local i = 0; i <= 10000; ++i)
{
    local stmt = db.QueryF("SELECT * FROM [Accounts] WHERE ID=%d;", i);

    if (!stmt.Step())
    {
        throw "this account does not exist";
    }

    acc_id = stmt.Get("ID").Integer;
    acc_name = stmt.Get("Name").String;
    acc_pass = stmt.Get("Pass").String;
    acc_active = stmt.Get("Active").Boolean;
}

StepBenchmark("Processing");

StopBenchmark(" SQLITE READ ");

Benchmark results:
[USR] ----------------------------- SQLITE READ ----------------------------
[USR]
[INF] Connection      :         304 microseconds
[INF] Processing      :      740621 microseconds
[INF] Total           :      740925 microseconds
[USR]
[USR] ----------------------------- SQLITE READ ----------------------------
Much slower than INI. 0.74 seconds compared to 0.17 seconds from INI.

Optimizations behind the scene:

• N/A

Bottlenecks behind the scene:

• Each query has to be compiled individually. It's the same as compiling your script 10,000 times for no reason at all.
• Others which will be discussed later.

In this benchmark I will use the same code as in the naive write approach except I will tell SQLite to simply hand-off the data to the OS for writing and then continue.

All I did was add this line of code after opening the database:
db.Exec("PRAGMA synchronous = OFF");
NOTE: Read about the dangers of PRAGMA synchronous OFF (http://sqlite.1065341.n5.nabble.com/How-dangerous-is-PRAGMA-Synchronous-OFF-td4250.html).

The resulting code was:
StartBenchmark(" SQLITE WRITE ");

local db = SQLite.Connection("test.db");

db.Exec("PRAGMA synchronous = OFF");

StepBenchmark("Connection");

db.Exec(@"CREATE TABLE IF NOT EXISTS [Accounts] (
    [ID] INTEGER  PRIMARY KEY NOT NULL,
    [Name] VARCHAR(32)  UNIQUE NOT NULL,
    [Pass] VARCHAR(128)  UNIQUE NOT NULL,
    [Active] INTEGER DEFAULT 0 NULL
);");

StepBenchmark("Creation");

for (local i = 0; i <= 10000; ++i)
{
    db.ExecF("INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (%d, 'user_%d', 'password_%d', %d);"
                i, i, i, i % 2);
}

StepBenchmark("Insertion");

StopBenchmark(" SQLITE WRITE ");

Benchmark results:
[USR] ---------------------------- SQLITE WRITE ----------------------------
[USR]
[INF] Connection      :         865 microseconds
[INF] Creation        :        1486 microseconds
[INF] Insertion       :     4442478 microseconds
[INF] Total           :     4444829 microseconds
[USR]
[USR] ---------------------------- SQLITE WRITE ----------------------------
Much better. Right? But still behind INI. 4.50 seconds is way more than 0.40 seconds.

---

In this benchmark I will take advantage of transactions to write large amounts of data in bulk. But also keep the previous optimization. This optimization does not apply to regular code where you barely use 1-2 insert queries every second. But let's try to out-perform the INI bulk write speeds.

StartBenchmark(" SQLITE WRITE ");

local db = SQLite.Connection("test.db");

db.Exec("PRAGMA synchronous = OFF");

StepBenchmark("Connection");

db.Exec(@"CREATE TABLE IF NOT EXISTS [Accounts] (
    [ID] INTEGER  PRIMARY KEY NOT NULL,
    [Name] VARCHAR(32)  UNIQUE NOT NULL,
    [Pass] VARCHAR(128)  UNIQUE NOT NULL,
    [Active] INTEGER DEFAULT 0 NULL
);");

StepBenchmark("Creation");

{
    local sqtrans = SQLite.Transaction(db);

    for (local i = 0; i <= 10000; ++i)
    {
        db.ExecF("INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (%d, 'user_%d', 'password_%d', %d);"
                    i, i, i, i % 2);
    }

    sqtrans.Commit();
}

StepBenchmark("Insertion");

StopBenchmark(" SQLITE WRITE ");

If you notice all I did was to wrap the loop in a single transaction.

Benchmark results:
[USR] ---------------------------- SQLITE WRITE ----------------------------
[USR]
[INF] Connection      :         748 microseconds
[INF] Creation        :        1574 microseconds
[INF] Insertion       :      210182 microseconds
[INF] Total           :      212504 microseconds
[USR]
[USR] ---------------------------- SQLITE WRITE ----------------------------
Who's the winner now? 0.21 seconds compared to 0.40 seconds for INI.

---

In this benchmark I will extend the previous benchmark by using a pre-compiled statement. As I've mentioned before SQLite has to compile that query each time you execute it. But wouldn't it be nice if we could compile that once and just updating the values we want to insert?

All I did this time was to create the statement once:
    local stmt = SQLite.Statement(db, "INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (?, ?, ?, ?);");
And then simply updating the values that I want to insert:
        stmt.Reset();
        stmt.IBindV(1, i);
        stmt.IBindS(2, "user_"+i);
        stmt.IBindS(3, "password_"+i);
        stmt.IBindV(4, i % 2);
        stmt.Step();
NOTE: This is also the most efficient method of avoiding SQL injection on your server.

The resulting code was:
StartBenchmark(" SQLITE WRITE ");

local db = SQLite.Connection("test.db");

db.Exec("PRAGMA synchronous = OFF");

StepBenchmark("Connection");

db.Exec(@"CREATE TABLE IF NOT EXISTS [Accounts] (
    [ID] INTEGER  PRIMARY KEY NOT NULL,
    [Name] VARCHAR(32)  UNIQUE NOT NULL,
    [Pass] VARCHAR(128)  UNIQUE NOT NULL,
    [Active] INTEGER DEFAULT 0 NULL
);");

StepBenchmark("Creation");

{
    local stmt = SQLite.Statement(db, "INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (?, ?, ?, ?);");
    local sqtrans = SQLite.Transaction(db);

    for (local i = 0; i <= 10000; ++i)
    {
        stmt.Reset();
        stmt.IBindV(1, i);
        stmt.IBindS(2, "user_"+i);
        stmt.IBindS(3, "password_"+i);
        stmt.IBindV(4, i % 2);
        stmt.Step();
    }

    sqtrans.Commit();
}

StepBenchmark("Insertion");

StopBenchmark(" SQLITE WRITE ");

Benchmark results:
[USR] ---------------------------- SQLITE WRITE ----------------------------
[USR]
[INF] Connection      :         760 microseconds
[INF] Creation        :        1563 microseconds
[INF] Insertion       :      139581 microseconds
[INF] Total           :      141904 microseconds
[USR]
[USR] ---------------------------- SQLITE WRITE ----------------------------
Seems we could go even faster than INI. This time the whole operation took around 0.13 seconds compared to the 0.40 seconds from INI.

---

In this benchmark I will try to go even further by taking a small risk. I will attempt to force the rollback journal to be stored in memory.

All I did this time was to add this line after opening the database:
db.Exec("PRAGMA journal_mode = MEMORY");
NOTE: This has the disadvantage that  if you lose power or your program crashes during a transaction your database could be left in a corrupt state with a partially-completed transaction. Normally, the plugin tries to avoid crashes as much as possible and in most cases the transaction is rolled back automatically if something failed. But still, this approach has a red flag on it.

The resulting code was:
StartBenchmark(" SQLITE WRITE ");

local db = SQLite.Connection("test.db");

db.Exec("PRAGMA synchronous = OFF");
db.Exec("PRAGMA journal_mode = MEMORY");

StepBenchmark("Connection");

db.Exec(@"CREATE TABLE IF NOT EXISTS [Accounts] (
    [ID] INTEGER  PRIMARY KEY NOT NULL,
    [Name] VARCHAR(32)  UNIQUE NOT NULL,
    [Pass] VARCHAR(128)  UNIQUE NOT NULL,
    [Active] INTEGER DEFAULT 0 NULL
);");

StepBenchmark("Creation");

{
    local stmt = SQLite.Statement(db, "INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (?, ?, ?, ?);");
    local sqtrans = SQLite.Transaction(db);

    for (local i = 0; i <= 10000; ++i)
    {
        stmt.Reset();
        stmt.IBindV(1, i);
        stmt.IBindS(2, "user_"+i);
        stmt.IBindS(3, "password_"+i);
        stmt.IBindV(4, i % 2);
        stmt.Step();
    }

    sqtrans.Commit();
}

StepBenchmark("Insertion");

StopBenchmark(" SQLITE WRITE ");

Benchmark results:
[USR] ---------------------------- SQLITE WRITE ----------------------------
[USR]
[INF] Connection      :         753 microseconds
[INF] Creation        :        1199 microseconds
[INF] Insertion       :      139712 microseconds
[INF] Total           :      141664 microseconds
[USR]
[USR] ---------------------------- SQLITE WRITE ----------------------------
Unfortunately this was as much as it could be optimized. On my system it didn't have any benefits at all. Simply because everything was executed in a single transaction. You need a few more transactions to see the benefits. But on a system with slow IO speeds. This will definitely start to show up no mater the number of transactions.

---

So there you go folks. If you truly want to optimize something you should know that there are always ways to do it. You just happen to don't know about them or simply ignore all forms of optimizations as I've seen some of you do very frequently. In the next post I'll go into detail about what SQLite allows you to do and INI doesn't.

In this benchmark I will try to optimize the read operation so that we can match the one of the INI benchmark or even obtain better results. First thing that comes to mind is to use pre-compiled statements like we did in the write benchmarks.

All I did was to created the statement once:
local stmt = SQLite.Statement(db, "SELECT * FROM [Accounts] WHERE ID=?;");
And bind the values that I want to work with:
    stmt.Reset();
    stmt.IBindV(1, i);
NOTE: This is also the most efficient method of avoiding SQL injection on your server.

The resulting code was:
StartBenchmark(" SQLITE READ ");

local db = SQLite.Connection("test.db");

StepBenchmark("Connection");

local acc_id, acc_name, acc_pass, acc_active;

local stmt = SQLite.Statement(db, "SELECT * FROM [Accounts] WHERE ID=?;");

for (local i = 0; i <= 10000; ++i)
{
    stmt.Reset();
    stmt.IBindV(1, i);

    if (!stmt.Step())
    {
        throw "this account does not exist";
    }

    acc_id = stmt.Get("ID").Integer;
    acc_name = stmt.Get("Name").String;
    acc_pass = stmt.Get("Pass").String;
    acc_active = stmt.Get("Active").Boolean;
}

StepBenchmark("Processing");

StopBenchmark(" SQLITE READ ");

Benchmark result:
[USR] ----------------------------- SQLITE READ ----------------------------
[USR]
[INF] Connection      :         226 microseconds
[INF] Processing      :      521106 microseconds
[INF] Total           :      521332 microseconds
[USR]
[USR] ----------------------------- SQLITE READ ----------------------------
Still slower than INI. 0.52 seconds is much slower than 0.17 seconds.

---

In this benchmark I'll be extending on the previous benchmark by removing the need to search for column names on each iteration. We know that in our loop we are going to retrieve the same columns over and over and over. Wouldn't be nice if we could retrieve them once and use them continuously after without having to run a search every time?

All I did was to retrieve the columns once outside the loop:
local col_id = stmt.Get("ID")
    , col_name = stmt.Get("Name")
    , col_pass = stmt.Get("Pass")
    , col_active = stmt.Get("Active");
And then use them whenever I want to query information about the current row in the statement:
    acc_id = col_id.Integer;
    acc_name = col_name.String;
    acc_pass = col_pass.String;
    acc_active = col_active.Boolean;
The resulting code was:
tBenchmark(" SQLITE READ ");

local db = SQLite.Connection("test.db");

StepBenchmark("Connection");

local acc_id, acc_name, acc_pass, acc_active;

local stmt = SQLite.Statement(db, "SELECT * FROM [Accounts] WHERE ID=?;");

local col_id = stmt.Get("ID")
    , col_name = stmt.Get("Name")
    , col_pass = stmt.Get("Pass")
    , col_active = stmt.Get("Active");

for (local i = 0; i <= 10000; ++i)
{
    stmt.Reset();
    stmt.IBindV(1, i);

    if (!stmt.Step())
    {
        throw "this account does not exist";
    }

    acc_id = col_id.Integer;
    acc_name = col_name.String;
    acc_pass = col_pass.String;
    acc_active = col_active.Boolean;
}

StepBenchmark("Processing");

StopBenchmark(" SQLITE READ ");

Benchmark result:
[USR] ----------------------------- SQLITE READ ----------------------------
[USR]
[INF] Connection      :         238 microseconds
[INF] Processing      :      429721 microseconds
[INF] Total           :      429959 microseconds
[USR]
[USR] ----------------------------- SQLITE READ ----------------------------
Still behind the INI. 0.42 is still slower than 0.17 we had with INI.

---

In this benchmark I'll be extending on the previous code by removing the need to query the database for each account. This code also shows what you can do with SQLite but not with INI. Which is to retrieve data in bulk. Of course, this is not always the case with most programs but just to show what SQLite can do in these kinds of situations.

All I did was to remove the where clause from the query:
local stmt = SQLite.Statement(db, "SELECT * FROM [Accounts];");
And also use a different kind of loop to iterate the selected data:
while (stmt.Step())
{
    acc_id = col_id.Integer;
    acc_name = col_name.String;
    acc_pass = col_pass.String;
    acc_active = col_active.Boolean;
}
The resulting code was:
StartBenchmark(" SQLITE READ ");

local db = SQLite.Connection("test.db");

StepBenchmark("Connection");

local acc_id, acc_name, acc_pass, acc_active;

local stmt = SQLite.Statement(db, "SELECT * FROM [Accounts];");

local col_id = stmt.Get("ID")
    , col_name = stmt.Get("Name")
    , col_pass = stmt.Get("Pass")
    , col_active = stmt.Get("Active");

while (stmt.Step())
{
    acc_id = col_id.Integer;
    acc_name = col_name.String;
    acc_pass = col_pass.String;
    acc_active = col_active.Boolean;
}

StepBenchmark("Processing");

if (acc_id != 10000)
{
    throw "there was an issue with the query";
}

StopBenchmark(" SQLITE READ ");

Benchmark results:
[USR] ----------------------------- SQLITE READ ----------------------------
[USR]
[INF] Connection      :         245 microseconds
[INF] Processing      :       54239 microseconds
[INF] Total           :       54484 microseconds
[USR]
[USR] ----------------------------- SQLITE READ ----------------------------
There we go. Now we're starting to see the true power of SQLite. 0.054 seconds is a much better result than 0.17 seconds from INI.

---

So there you go folks. Again demonstrating that if you truly want to optimize something, you can definitely achieve better results. It all depends on the situation that you're in. In the next post I'll take the current SQLite benchmarks to extreme by using in memory databases to show how much disk IO impacts performance.

The reason I've created this benchmark is because the INI file is stored in memory and I wanted to see a true one on one comparison. In this benchmark I'll be combining the code from from write optimization #3 and read optimization #2. Which means that I'll be searching the database for each individual account.

The resulting code will be:
StartBenchmark(" SQLITE MEMORY READ/WRITE ");

local db = SQLite.Connection(":memory:");

db.Exec("PRAGMA synchronous = OFF");

StepBenchmark("Connection");

db.Exec(@"CREATE TABLE IF NOT EXISTS [Accounts] (
    [ID] INTEGER  PRIMARY KEY NOT NULL,
    [Name] VARCHAR(32)  UNIQUE NOT NULL,
    [Pass] VARCHAR(128)  UNIQUE NOT NULL,
    [Active] INTEGER DEFAULT 0 NULL
);");

StepBenchmark("Creation");

{
    local stmt = SQLite.Statement(db, "INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (?, ?, ?, ?);");
    local sqtrans = SQLite.Transaction(db);

    for (local i = 0; i <= 10000; ++i)
    {
        stmt.Reset();
        stmt.IBindV(1, i);
        stmt.IBindS(2, "user_"+i);
        stmt.IBindS(3, "password_"+i);
        stmt.IBindV(4, i % 2);
        stmt.Step();
    }

    sqtrans.Commit();
}

StepBenchmark("Insertion");

local acc_id, acc_name, acc_pass, acc_active;

local stmt = SQLite.Statement(db, "SELECT * FROM [Accounts] WHERE ID=?;");

local col_id = stmt.Get("ID")
    , col_name = stmt.Get("Name")
    , col_pass = stmt.Get("Pass")
    , col_active = stmt.Get("Active");

for (local i = 0; i <= 10000; ++i)
{
    stmt.Reset();
    stmt.IBindV(1, i);

    if (!stmt.Step())
    {
        throw "this account does not exist";
    }

    acc_id = col_id.Integer;
    acc_name = col_name.String;
    acc_pass = col_pass.String;
    acc_active = col_active.Boolean;
}

StepBenchmark("Processing");

StopBenchmark(" SQLITE MEMORY READ/WRITE ");

Benchmark results:
[USR] ---------------------- SQLITE MEMORY READ/WRITE ----------------------
[USR]
[INF] Connection      :         319 microseconds
[INF] Creation        :        1578 microseconds
[INF] Insertion       :      137226 microseconds
[INF] Processing      :       83858 microseconds
[INF] Total           :      222981 microseconds
[USR]
[USR] ---------------------- SQLITE MEMORY READ/WRITE ----------------------
If you notice something funny. The write speed still hasn't changed from 0.13 seconds. Which means that the real bottleneck here was the actual Squirrel code. Whereas the read code decreased from 0.42 seconds to a mere 0.083 seconds. And if you remember, this read code was not the fastest one.

---

In this benchmark I'll be expanding on the previous code except I'll be using the code from optimized read #3 which reads the entire data at once.

The resulting code was:
StartBenchmark(" SQLITE MEMORY READ/WRITE ");

local db = SQLite.Connection(":memory:");

db.Exec("PRAGMA synchronous = OFF");

StepBenchmark("Connection");

db.Exec(@"CREATE TABLE IF NOT EXISTS [Accounts] (
    [ID] INTEGER  PRIMARY KEY NOT NULL,
    [Name] VARCHAR(32)  UNIQUE NOT NULL,
    [Pass] VARCHAR(128)  UNIQUE NOT NULL,
    [Active] INTEGER DEFAULT 0 NULL
);");

StepBenchmark("Creation");

{
    local stmt = SQLite.Statement(db, "INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (?, ?, ?, ?);");
    local sqtrans = SQLite.Transaction(db);

    for (local i = 0; i <= 10000; ++i)
    {
        stmt.Reset();
        stmt.IBindV(1, i);
        stmt.IBindS(2, "user_"+i);
        stmt.IBindS(3, "password_"+i);
        stmt.IBindV(4, i % 2);
        stmt.Step();
    }

    sqtrans.Commit();
}

StepBenchmark("Insertion");

local acc_id, acc_name, acc_pass, acc_active;

local stmt = SQLite.Statement(db, "SELECT * FROM [Accounts];");

local col_id = stmt.Get("ID")
    , col_name = stmt.Get("Name")
    , col_pass = stmt.Get("Pass")
    , col_active = stmt.Get("Active");

while (stmt.Step())
{
    acc_id = col_id.Integer;
    acc_name = col_name.String;
    acc_pass = col_pass.String;
    acc_active = col_active.Boolean;
}

StepBenchmark("Processing");

StopBenchmark(" SQLITE MEMORY READ/WRITE ");

Benchmark results:
[USR] ---------------------- SQLITE MEMORY READ/WRITE ----------------------
[USR]
[INF] Connection      :         324 microseconds
[INF] Creation        :        1113 microseconds
[INF] Insertion       :      137058 microseconds
[INF] Processing      :       49567 microseconds
[INF] Total           :      188062 microseconds
[USR]
[USR] ---------------------- SQLITE MEMORY READ/WRITE ----------------------
Again, the same write speed. And a small bump in read speed from 0.054 seconds to 0.049 seconds. Which means, that we've reached the point where the actual Squirrel code is the bottleneck.

In this benchmark I'll be extending on the code from the write optimization #2 which took about 0.21 seconds to execute and required explicit use of transactions. The reason I've created this benchmark is to demonstrate the effectiveness of the queue implementation that the plugin/module comes with. Internally this uses a transaction to send all or a portion of the queued queries. Queued queries are never executed when created. Instead, they're executed when you want and they're executed in a single transaction.

The resulting code was:
StartBenchmark(" SQLITE WRITE ");

local db = SQLite.Connection("test.db");

db.Exec("PRAGMA synchronous = OFF");

StepBenchmark("Connection");

db.Exec(@"CREATE TABLE IF NOT EXISTS [Accounts] (
    [ID] INTEGER  PRIMARY KEY NOT NULL,
    [Name] VARCHAR(32)  UNIQUE NOT NULL,
    [Pass] VARCHAR(128)  UNIQUE NOT NULL,
    [Active] INTEGER DEFAULT 0 NULL
);");

StepBenchmark("Creation");

// Allocate space upfront for 10000 queries
db.ReserveQueue(10000);

StepBenchmark("Allocation");

for (local i = 0; i <= 10000; ++i)
{
    db.QueueF("INSERT INTO [Accounts] (ID, Name, Pass, Active) VALUES (%d, 'user_%d', 'password_%d', %d);"
                i, i, i, i % 2);
}

StepBenchmark("Queue");

db.Flush(this, function(status, query)
{
    // Log the incident
    SqLog.Err("Failed to flush query: %s", query);
    SqLog.Inf("=> Reason: %s", db.ErrMsg);
    // Continue flushing the remaining queries
    return true;
});

StepBenchmark("Flush");

StopBenchmark(" SQLITE WRITE ");

Benchmark result:
[USR] ---------------------------- SQLITE WRITE ----------------------------
[USR]
[INF] Connection      :         770 microseconds
[INF] Creation        :        1478 microseconds
[INF] Allocation      :         817 microseconds
[INF] Queue           :       18525 microseconds
[INF] Flush           :      176447 microseconds
[INF] Total           :      198037 microseconds
[USR]
[USR] ---------------------------- SQLITE WRITE ----------------------------
It's not a significant performance bump to just 0.17 seconds plus the time to queue them. However, if you take into account the fact that each query has it's own transaction. In real code when you need to execute a query here and there. This approach will have a significant impact.

---

This approach is best used with a routine that flushes a certain amount of queries at certain intervals. For example the following code flushes 10 queries every second:
SqRoutine.Create(this, function(num)
{
    db.Flush(num, this, function(status, query)
    {
        // Log the incident
        SqLog.Err("Failed to flush query: %s", query);
        SqLog.Inf("=> Reason: %s", db.ErrMsg);
        // Continue flushing the remaining queries
        return true;
    });
}, 1000, 0, 10).SetTag("DatabaseFlush");
Or the following which flushes all queries every 4 seconds;
SqRoutine.Create(this, function()
{
    db.Flush(this, function(status, query)
    {
        // Log the incident
        SqLog.Err("Failed to flush query: %s", query);
        SqLog.Inf("=> Reason: %s", db.ErrMsg);
        // Continue flushing the remaining queries
        return true;
    });
}, 4000, 0).SetTag("DatabaseFlush");
It's just an example of what can be achieved.

Implemented a TCC module tonight to generate new squirrel API at runtime or to move performance critical code to native machine code if Squirrel is lagging.

I am looking for a decent host for the wiki. The one that I'm currently using might not be stable for long. The host only needs PHP (no database) and FTP access because the documentation is stored in plain text. And FTP is needed for making bulk changes.

If you have any suggestions for a decent stable host then feel free to let me know. Preferably by PM.

I am closing this topic because the plug-in is quite stable for use in game-mods. Even though the plug-in is in beta, it is quite stable. There isn't too much documentation because I can't maintain the plug-in and documentation at the same time. The plug-in expects someone that can actually read his way through the source code (quite simple once you get the hang of it). Future documentation will be available here (http://sqmod.is-great.org/doku.php). It's a temporary host for now.

The plug-in itself offers at least as much as the official. Actually, a lot more than that.

The SQLite module is probable the most complete module out of all.
The IRC follows being also complete. Currently waiting to implement a way of adding colors to messages.
The XML module is quite functional but missing some way of iterating on document nodes.
The JSON module is still a work in progress.
The MySQL module is also a work in progress. Works but offers only primitive ways of retrieving data.
The MaxmindDB module is also a work in progress. Not even functional and should not be used or attempted to compile.
The Mongoose module is also a work in progress. Not even functional and should not be used or attempted to compile.

Please note that this is quite an advanced scripting plugin. If you've never experienced with higher level programming concepts then this plug-in is not for you. This plug-in is meant for people that want more than what the official plug-in provides and tries to address all the issues and limitations currently found in the official plug-in.

The plug-in is provided in such a way that it's easy for anyone to compile it. On windows, all you need is mingw-w64 (https://sourceforge.net/projects/mingw-w64/) and code-blocks (http://www.codeblocks.org/) and on Linux you pretty much use gcc instead of mingw.

If you cannot build binaries yourself then you can probably ask me or @Drake on #SLC @LUNet A topic with binaries will be made when an official release is made after all modules are complete.

Many thanks to @Drake helping out with testing the plug-in and being an early adopter.

Plug-in repository: https://github.com/iSLC/VCMP-SqMod