Vice City: Multiplayer

VC:MP Discussion => General Discussion => Topic started by: ysc3839 on Feb 03, 2016, 05:57 AM

Title: Unicode support in VCMP
Post by: ysc3839 on Feb 03, 2016, 05:57 AM
I'm trying to wirte a plugin that hooks VCMP's draw text function to make VCMP supports Unicode. And I will open the source code after I finished. :)
Now I wonder if this is allowed? @Stormeus
Title: Re: Unicode support in VCMP
Post by: Murdock on Feb 03, 2016, 06:26 AM
As long you won't show how to actually inject other dll files in the game or make it possible for someone to load a different file, because that wouldn't be appreciated.

Not speaking on behalf of Stormeus, just giving my opinion
Title: Re: Unicode support in VCMP
Post by: KakaroT on Feb 03, 2016, 06:38 AM
Quote from: Murdock on Feb 03, 2016, 06:26 AMAs long you won't show how to actually inject other dll files in the game or make it possible for someone to load a different file, because that wouldn't be appreciated.

Not speaking on behalf of Stormeus, just giving my opinion
Yes. Or we gonna have bunch of more hacks right after you show how to do it.
Title: Re: Unicode support in VCMP
Post by: Thijn on Feb 03, 2016, 06:51 AM
I'm wondering how you plan on sending the chat to the server and keep clients without the plugin compatible.
Title: Re: Unicode support in VCMP
Post by: . on Feb 03, 2016, 06:54 AM
Quote from: Thijn on Feb 03, 2016, 06:51 AMI'm wondering how you plan on sending the chat to the server and keep clients without the plugin compatible.

Ha, that's going to be a tall wall.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 03, 2016, 06:55 AM
Quote from: Murdock on Feb 03, 2016, 06:26 AMAs long you won't show how to actually inject other dll files in the game or make it possible for someone to load a different file, because that wouldn't be appreciated.

Not speaking on behalf of Stormeus, just giving my opinion
Do you mean you want me to tell you how this plugin inject into the game? Currently this plugins is unfinished, I use a loader write by myself to test it. I will find a good way for users in the future. :D
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 03, 2016, 06:58 AM
Quote from: Thijn on Feb 03, 2016, 06:51 AMI'm wondering how you plan on sending the chat to the server and keep clients without the plugin compatible.
I haven't considered typing Unicode char. Just show Unicode char. And it's really a big problem when clients haven't installed the plugin. I think server can let users change the language.
Title: Re: Unicode support in VCMP
Post by: Stormeus on Feb 03, 2016, 07:05 AM
Quote from: Stormeus on Feb 03, 2016, 07:03 AM
Quote from: ysc3839 on Feb 03, 2016, 06:20 AMI got the font bitmap from VCMP. You can see "á é í ó ú ä ë ï ö ü" in it. Why not support them?

Because the client uses the Windows-1252 codepage for the font bitmap whereas the server does not uses CP1252 so the server's character mappings won't map to the client's if we don't restrict everything to ANSI.

Kindly do not do this. Full Unicode is a tall order in itself and there are many characters in it which can break text rendering entirely and probably fuck up the server.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 03, 2016, 07:14 AM
Quote from: Stormeus on Feb 03, 2016, 07:05 AM
Quote from: Stormeus on Feb 03, 2016, 07:03 AM
Quote from: ysc3839 on Feb 03, 2016, 06:20 AMI got the font bitmap from VCMP. You can see "á é í ó ú ä ë ï ö ü" in it. Why not support them?

Because the client uses the Windows-1252 codepage for the font bitmap whereas the server does not uses CP1252 so the server's character mappings won't map to the client's if we don't restrict everything to ANSI.

Kindly do not do this. Full Unicode is a tall order in itself and there are many characters in it which can break text rendering entirely and probably fuck up the server.
I want to have a try.
Title: Re: Unicode support in VCMP
Post by: Stormeus on Feb 03, 2016, 07:16 AM
Go for it, if you somehow get it to work on all fronts then I'd be impressed, though even if it does work it would definitely not be the ideal approach.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 03, 2016, 07:35 AM
Quote from: Stormeus on Feb 03, 2016, 07:16 AMGo for it, if you somehow get it to work on all fronts then I'd be impressed, though even if it does work it would definitely not be the ideal approach.
Thanks for you understanding! ;D
Title: Re: Unicode support in VCMP
Post by: KAKAN on Feb 03, 2016, 08:50 AM
Quote from: Stormeus on Feb 03, 2016, 07:16 AMGo for it, if you somehow get it to work on all fronts then I'd be impressed, though even if it does work it would definitely not be the ideal approach.
Yeah, that would be great if we add Unicode support in VCMP. We can add different languages :)
Title: Re: Unicode support in VCMP
Post by: Stormeus on Feb 03, 2016, 07:39 PM
Quote from: KAKAN on Feb 03, 2016, 08:50 AM
Quote from: Stormeus on Feb 03, 2016, 07:16 AMGo for it, if you somehow get it to work on all fronts then I'd be impressed, though even if it does work it would definitely not be the ideal approach.
Yeah, that would be great if we add Unicode support in VCMP. We can add different languages :)
Quote from: Stormeus on Feb 03, 2016, 04:23 AMWe'd mostly have to redo the way that we draw text since we're currently just using bitmaps, and a Unicode bitmap would be gigantic — too big to store in memory. There'd also be some really annoying netcode changes.

It wouldn't be impossible; I've tried experimenting with FreeType but I've yet to get that working.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 05, 2016, 07:46 AM
The first screenshot! (https://forum.vc-mp.org/proxy.php?request=http%3A%2F%2Fi64.tinypic.com%2F28cogno.png&hash=083bfb54036d54a7b53226a418867563ec43ae8c)
Also I wonder hot to make it like VCMP's text.
Title: Re: Unicode support in VCMP
Post by: Anik on Feb 05, 2016, 08:18 AM
Will you add all kind of languages?
Title: Re: Unicode support in VCMP
Post by: KAKAN on Feb 05, 2016, 08:44 AM
Quote from: [DS]Anik on Feb 05, 2016, 08:18 AMWill you add all kind of languages?

Maybe, but if he does, we can make a server containing 10+ langs :DDD
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 05, 2016, 09:00 AM
Quote from: [DS]Anik on Feb 05, 2016, 08:18 AMWill you add all kind of languages?
This plugin will use UTF-8 charset so that you can use all langages! ;)
Title: Re: Unicode support in VCMP
Post by: Anik on Feb 05, 2016, 09:12 AM
Quote from: KAKAN on Feb 05, 2016, 08:44 AM
Quote from: [DS]Anik on Feb 05, 2016, 08:18 AMWill you add all kind of languages?

Maybe, but if he does, we can make a server containing 10+ langs :DDD
:D :D :D
Quote from: ysc3839 on Feb 05, 2016, 09:00 AM
Quote from: [DS]Anik on Feb 05, 2016, 08:18 AMWill you add all kind of languages?
This plugin will use UTF-8 charset so that you can use all langages! ;)
Hmm, How much time would it take to complete?
Title: Re: Unicode support in VCMP
Post by: EK.IceFlake on Feb 05, 2016, 09:42 AM
Great!
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 05, 2016, 09:58 AM
Quote from: [DS]Anik on Feb 05, 2016, 09:12 AM
Quote from: KAKAN on Feb 05, 2016, 08:44 AM
Quote from: [DS]Anik on Feb 05, 2016, 08:18 AMWill you add all kind of languages?

Maybe, but if he does, we can make a server containing 10+ langs :DDD
:D :D :D
Quote from: ysc3839 on Feb 05, 2016, 09:00 AM
Quote from: [DS]Anik on Feb 05, 2016, 08:18 AMWill you add all kind of languages?
This plugin will use UTF-8 charset so that you can use all langages! ;)
Hmm, How much time would it take to complete?
If you like, you can use this version. :P But the text style is not like VCMP. I want to improve it. Also, if you go back to desktop and return to the game, the game will crash. I'm trying to fix it.
Title: Re: Unicode support in VCMP
Post by: EK.IceFlake on Feb 05, 2016, 10:51 AM
Quote from: Murdock on Feb 03, 2016, 06:26 AMAs long you won't show how to actually inject other dll files in the game or make it possible for someone to load a different file, because that wouldn't be appreciated.

Not speaking on behalf of Stormeus, just giving my opinion
Which is just what I did with Georges browser
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 05, 2016, 11:30 AM
Quote from: NE.CrystalBlue on Feb 05, 2016, 10:51 AM
Quote from: Murdock on Feb 03, 2016, 06:26 AMAs long you won't show how to actually inject other dll files in the game or make it possible for someone to load a different file, because that wouldn't be appreciated.

Not speaking on behalf of Stormeus, just giving my opinion
Which is just what I did with Georges browser
It's easier if VCMP allows loading asi files. Now I think I can rename the original VCMP dll to "vcmp-game.dll.1", and rename this plugin to "vcmp-game.dll". Then the plugin loads the original VCMP dll and hooks it.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 11, 2016, 05:13 PM
The first source code! I named it "vcmp-i18n". This version is under development. It isn't for users, just for developers to review.

https://bitbucket.org/ysc3839/vcmp-i18n

Known issues: 1) No Unicode support.
2) When resize the window, the game will crash.

Also, I tried many ways to make it like VCMP's style, but I can't. @Stormeus could you give me some help?
Title: Re: Unicode support in VCMP
Post by: TheMallard on Feb 12, 2016, 10:26 AM
It will be great, if you will continue developing this plugin!
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 12, 2016, 04:39 PM
Quote from: ysc3839 on Feb 11, 2016, 05:13 PMThe first source code! I named it "vcmp-i18n". This version is under development. It isn't for users, just for developers to review.

https://bitbucket.org/ysc3839/vcmp-i18n

Known issues: 1) No Unicode support.
2) When resize the window, the game will crash.

Also, I tried many ways to make it like VCMP's style, but I can't. @Stormeus could you give me some help?
@Stormeus Have you seen this? Please reply soon. :)
Title: Re: Unicode support in VCMP
Post by: Stormeus on Feb 12, 2016, 07:12 PM
The game is crashing because you haven't handled D3D's device lost and reset events, so when that happens you end up doing operations on a D3DDEVICE that doesn't exist anymore and crash the game. Also not sure what you mean by VC:MP style since it's pretty much just the same text drawn slightly larger in black underneath.

While this does solve the problem to an extent, there are performance drawbacks to calling DrawText on every frame, hence why we use bitmaps in the first place.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 12, 2016, 07:26 PM
Quote from: Stormeus on Feb 12, 2016, 07:12 PMThe game is crashing because you haven't handled D3D's device lost and reset events, so when that happens you end up doing operations on a D3DDEVICE that doesn't exist anymore and crash the game. Also not sure what you mean by VC:MP style since it's pretty much just the same text drawn slightly larger in black underneath.

While this does solve the problem to an extent, there are performance drawbacks to calling DrawText on every frame, hence why we use bitmaps in the first place.
Thanks! :)
See this picture (https://forum.vc-mp.org/proxy.php?request=http%3A%2F%2Fi64.tinypic.com%2F28cogno.png&hash=083bfb54036d54a7b53226a418867563ec43ae8c)
Left is VC:MP's style, right isn't.
I also found DrawText will decrease the fps, especially on my laptop. I'm considering use FreeType. I think it may better if there is a cache bitmap for chat box.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Feb 13, 2016, 04:51 PM
Added UTF-8 support. Now it can use VC:MP builtin font for some characters.
(https://forum.vc-mp.org/proxy.php?request=http%3A%2F%2Fi65.tinypic.com%2Fmmcv29.png&hash=c0d55b69974a56448544e486e862adccec3f0050)
Title: Re: Unicode support in VCMP
Post by: Stormeus on Feb 13, 2016, 06:08 PM
That's still just extended Latin on the in-built font, though.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on May 01, 2016, 07:44 AM
You may want to know whether I will support GUI texts. Now I want to say it's difficult to do that. Now I'm working on the new VC:MP browser. I'll update this when I finished the browser.
Title: Re: Unicode support in VCMP
Post by: EightyVice on May 01, 2016, 04:47 PM
Good work @ysc3839
You surprised me by your work :)
Title: Re: Unicode support in VCMP
Post by: RW on May 08, 2016, 03:28 AM
Great! We will use them in the future!终于可以愉快地说中文了! Finally we can speak Chinese. LOL
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Jul 18, 2016, 01:59 AM
The first useable version! :D
Only supports 04rel003 version 571C093D. Not support XP!!!
https://bitbucket.org/ysc3839/vcmp-i18n/downloads
Reanme the vcmp-game.dll to vcmp-game1.dll and rename vcmp-i18n-dwrite-dbg.dll to vcmp-game.dll.
Or inject vcmp-i18n-dwrite-dbg.dll to VCMP.
Screenshot:(https://s26.postimg.org/68i32h7k9/dwritetest4.png)
Title: Re: Unicode support in VCMP
Post by: Luis_Labarca on Jul 18, 2016, 02:21 AM
@ysc3839 good work bro I congratulate you ;)
Title: Re: Unicode support in VCMP
Post by: KAKAN on Jul 18, 2016, 08:40 AM
Awesome work mate!
Title: Re: Unicode support in VCMP
Post by: EightyVice on Jul 18, 2016, 09:54 AM
I like your way to support VCMP @ysc3839
Title: Re: Unicode support in VCMP
Post by: Miracle on Jul 18, 2016, 10:20 AM
Congratulations
Title: Re: Unicode support in VCMP
Post by: ℛḝξ☂ on Jul 18, 2016, 10:58 AM
keep it ;)
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Jul 18, 2016, 01:03 PM
Quote from: Rest on Jul 18, 2016, 10:58 AMkeep it ;)
I'll keep development if I have time. Or if @Stormeus gives me VCMP's font code. :D
Title: Re: Unicode support in VCMP
Post by: Stormeus on Jul 18, 2016, 01:24 PM
@ysc3839 The font code for VC:MP involves creating a bitmap of all Latin characters and storing it in a DirectX texture to blit later. Drawing via this method is fast but supporting Unicode with it is extremely memory exhaustive. Even if I were to give you the source, it'd be virtually useless for what you're trying to do.
Title: Re: Unicode support in VCMP
Post by: EK.IceFlake on Jul 18, 2016, 01:43 PM
Cant you generate your own bitmap where whenever a new letter is used you put it in your new bitmap. If it was used previously then draw it from the bitmap otherwise read it from the font and add it to your bitmap
Title: Re: Unicode support in VCMP
Post by: EK.IceFlake on Jul 18, 2016, 01:45 PM
Quote from: ysc3839 on Feb 11, 2016, 05:13 PMThe first source code! I named it "vcmp-i18n". This version is under development. It isn't for users, just for developers to review.

https://bitbucket.org/ysc3839/vcmp-i18n

Known issues: 1) No Unicode support.
2) When resize the window, the game will crash.

Also, I tried many ways to make it like VCMP's style, but I can't. @Stormeus could you give me some help?
LOL
"Unicode support in VCMP"
"Known issues: 1) No Unicode support."
Title: Re: Unicode support in VCMP
Post by: Stormeus on Jul 18, 2016, 01:58 PM
Quote from: EK.CrystalBlue on Jul 18, 2016, 01:43 PMCant you generate your own bitmap where whenever a new letter is used you put it in your new bitmap. If it was used previously then draw it from the bitmap otherwise read it from the font and add it to your bitmap

That's an idea, except that introduces a denial of service attack where a player could just send a large string of Unicode characters to exhaust the memory of everyone in the server.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Jul 18, 2016, 02:35 PM
Quote from: Stormeus on Jul 18, 2016, 01:24 PM@ysc3839 The font code for VC:MP involves creating a bitmap of all Latin characters and storing it in a DirectX texture to blit later. Drawing via this method is fast but supporting Unicode with it is extremely memory exhaustive. Even if I were to give you the source, it'd be virtually useless for what you're trying to do.
I want to imitate VCMP's font style but after disassembled VCMP I still don't know how it draws. There's a function between pTexture->Lock and Unlock I don't understand how it works. So I need the source code.

Quote from: Stormeus on Jul 18, 2016, 01:58 PM
Quote from: EK.CrystalBlue on Jul 18, 2016, 01:43 PMCant you generate your own bitmap where whenever a new letter is used you put it in your new bitmap. If it was used previously then draw it from the bitmap otherwise read it from the font and add it to your bitmap

That's an idea, except that introduces a denial of service attack where a player could just send a large string of Unicode characters to exhaust the memory of everyone in the server.
About the performance. IIRC, MTA:SA have a font cache map that caches character bitmap recently used. And my thes version uses DirectWrite and it looks like very fast since it have hardware acceleration.

PS:MTA:SA have two ways to draw texts. One is use ID3DXFont. The other is use FreeType to read glyph bitmap and save to a map. If the map full then some glyph will be removed.

PPS: Informations for this test version. I uses DirectWrite and Direct2D to draw texts to a dc and copy the dc to a Direct3D texture。Then draw this texture to sprite. It's easy to imitate VCMP's font style using DirectWrite's custom text renderer and Direct2D's geometry.

PPPS: @Stormeus I want to blame that VCMP uses the SetRect function to set a RECT struct.
I think it's better to use:(I'm using VS2015 it supports this. IIRC this is supported since C++11)RECT rc = { l, t, r, b };
Title: Re: Unicode support in VCMP
Post by: MEGAMIND on Jul 18, 2016, 04:59 PM
Awesome work bro! salute u
Title: Re: Unicode support in VCMP
Post by: EK.IceFlake on Jul 18, 2016, 07:02 PM
Quote from: Stormeus on Jul 18, 2016, 01:58 PM
Quote from: EK.CrystalBlue on Jul 18, 2016, 01:43 PMCant you generate your own bitmap where whenever a new letter is used you put it in your new bitmap. If it was used previously then draw it from the bitmap otherwise read it from the font and add it to your bitmap

That's an idea, except that introduces a denial of service attack where a player could just send a large string of Unicode characters to exhaust the memory of everyone in the server.
You can purge the least used characters when some bitmap limit [maybe a megabyte] is reached
Title: Re: Unicode support in VCMP
Post by: Stormeus on Jul 18, 2016, 07:09 PM
Quote from: EK.CrystalBlue on Jul 18, 2016, 07:02 PM
Quote from: Stormeus on Jul 18, 2016, 01:58 PM
Quote from: EK.CrystalBlue on Jul 18, 2016, 01:43 PMCant you generate your own bitmap where whenever a new letter is used you put it in your new bitmap. If it was used previously then draw it from the bitmap otherwise read it from the font and add it to your bitmap

That's an idea, except that introduces a denial of service attack where a player could just send a large string of Unicode characters to exhaust the memory of everyone in the server.
You can purge the least used characters when some bitmap limit [maybe a megabyte] is reached

That doesn't really solve the issue though because someone could just forge a chat message packet and fill it with as many Unicode bytes as possible, which would exhaust memory anyway. Setting a limit like 1MB makes no sense because I'm pretty sure the bitmap of pure Latin characters takes up more than that at the moment anyway.
Title: Re: Unicode support in VCMP
Post by: . on Jul 18, 2016, 07:14 PM
Quote from: EK.CrystalBlue on Jul 18, 2016, 07:02 PMYou can purge the least used characters when some bitmap limit [maybe a megabyte] is reached

So now we're supposed to keep track of which characters are used? That would imply scanning every string and processing each character, and for each character keep a time-stamp of when it was used and then periodically scan those time-stamps to see which is too old. And then remove the associated character from the bitmap. Then if someone uses that character, create the bitmap again... I mean come on... is this how the industry did this so far?
Title: Re: Unicode support in VCMP
Post by: KAKAN on Jul 19, 2016, 08:43 AM
why not do something like if the user's RAM is more than 2 GB, enable a option a use Unicode chars. And if it's <2 GB, ask the user about it, also, create a function in the settings menu :D
Title: Re: Unicode support in VCMP
Post by: EK.IceFlake on Jul 19, 2016, 11:03 AM
Quote from: . on Jul 18, 2016, 07:14 PM
Quote from: EK.CrystalBlue on Jul 18, 2016, 07:02 PMYou can purge the least used characters when some bitmap limit [maybe a megabyte] is reached

So now we're supposed to keep track of which characters are used? That would imply scanning every string and processing each character, and for each character keep a time-stamp of when it was used and then periodically scan those time-stamps to see which is too old. And then remove the associated character from the bitmap. Then if someone uses that character, create the bitmap again... I mean come on... is this how the industry did this so far?
Except when you remember they are only a few thousand operations, which would take a few KHz of the processors few GHz, which only 1 GHz is 1024MHz which only 1 MHz is 1024KHz.
Title: Re: Unicode support in VCMP
Post by: Stormeus on Jul 19, 2016, 11:30 AM
We're not hacking Unicode into the bitmap system, full stop. The traditional approach generally involves using FreeType or DirectWrite to render the text and only once we have a satisfactory implementation of that will we migrate to Unicode.
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Jul 19, 2016, 11:39 AM
Quote from: KAKAN on Jul 19, 2016, 08:43 AMwhy not do something like if the user's RAM is more than 2 GB, enable a option a use Unicode chars. And if it's <2 GB, ask the user about it, also, create a function in the settings menu :D
In Windows, 32bit programs can only access 2GB memory.
Title: Re: Unicode support in VCMP
Post by: EK.IceFlake on Jul 19, 2016, 11:55 AM
I have a question. Does the mGUI support Unicode?
Title: Re: Unicode support in VCMP
Post by: ysc3839 on Jul 19, 2016, 12:14 PM
Quote from: EK.CrystalBlue on Jul 19, 2016, 11:55 AMI have a question. Does the mGUI support Unicode?
As I know it doesn't.
Title: Re: Unicode support in VCMP
Post by: KAKAN on Jul 19, 2016, 01:36 PM
Quote from: ysc3839 on Jul 19, 2016, 11:39 AM
Quote from: KAKAN on Jul 19, 2016, 08:43 AMwhy not do something like if the user's RAM is more than 2 GB, enable a option a use Unicode chars. And if it's <2 GB, ask the user about it, also, create a function in the settings menu :D
In Windows, 32bit programs can only access 2GB memory.
They can access upto 4GB, I think so.