TCP/UDP Server

[Richard Kelly]

As I am now moving into the application specific portion of evolving my database server, I'd thought in a good idea to see if a small revival in the FF forums might develop with some fresh material.

The CTDBServ FF project attached is, at this point, a basic server shell. I've had fun including lots of code snippets gleaned from these forums over the past several years and even used Jose's cWindow class directly - twice!

To the best of my knowledge, I used a FF function every chance I could, otherwise, SDK api calls. The whole server, to this point only took 19 hours of effort over two weeks to get to this point. Without the FF IDE, I'd still be at it.

The design is a bit specific to my goals. It is, of course, multithreaded but is not multihomed. I have one listening request port and one UDP port I'll respond to if the message received is one I recognize and the response will be the IP and port that the TCP request port is listening to. This lets clients find the server and periodically check if it's still online. If the server is recycled back with a different listening IP/Port, clients can always get the "latest" way to connect for requests.

I decided to play nice in the Microsoft system going away scheme and support for the session end messages is included. I'd like to figure out the auto restart scheme to my satisfaction when it's not me that is shutting down the system. Of course, examples of the subclassing needed for the %WM_QUERYENDSESSION and %WM_ENDSESSION messages are right here where Paul, has over the years, sprinkled our threads with the FF fairy dust.

There is a basic challenge/response handshake on each new connection where both the server and client can satisfy the other is in possession of a shared pass phrase.

The activity listview is self trimming so we don't go crazy and suck up memory indefinitely and I couldn't resist the angelic calling for ownerdraw.

There is a trace log that can be started/stopped. Logs are built in the LOG subfolder from the EXE folder location.

I have stuck to a receive a request command, respond ready, receive some data, respond with some data paradigm. You could easily alter that if you want to.

I'm walking away from XP and this is my first breakaway effort which needs Server 2008/Vista SP1 or later to run.

Included in the attachment is a very basic CTDBClnt program to get you started.

There are some stubs for you to drop in the "database" of your choice. I'm going to start with SQLite3 and that is not a knock on SQLitening, a terrific and great tool. I'm just on a different mission.

Some of my "mission" goals are that all the actual SQL code resides with the server. Clients just ask their "question" without regard to how the answer for their request is handled. Additionally, the server also checks in with my web site and will deliver news, updates, tips, etc to each client on demand and handle other database tasks such as backup and any application background tasks that are scheduled, deliver email/SMS messages and create PDF reports to send back to each client. The basic framework for all this will be built into the server design. I shared my PDF object source code earlier this year on the PB forums and can't remember if I also posted it here.

Can you imagine the grinding in the PB forums if I uploaded a FF project there?

I don't claim the prototype, at this point, is rock solid. If you find my mistakes, please post back here as I will as testing moves forward. I have created a loop that ran 10,000 times just a few minutes ago with each iteration creating a thread that opened an connection with the server as a stress test and found no errors. I think that will exceed anything my clients will throw at it. The server ran on my machine with a direct network connection and the client from my wife's laptop over the wireless part of our home network. For giggles I took the laptop as far away in the house as I could....far side of the house up 2 stories.

Afrikaans - 'n Geseende Kersfees en 'n voorspoedige Nuwejaar
Afrikaans - Een Plesierige Kerfees
Albanian -- Gezuar Krishtlindje
Arabic - I'd Miilad Said Oua Sana Saida
Armenian - Shenoraavor Nor Dari yev Pari Gaghand
Azeri - Tezze Iliniz Yahsi Olsun
Basque - Zorionstsu Eguberri. Zoriontsu Urte Berri On
Bengali - Bodo Din Shubh Lamona
Bohemian - Vesele Vanoce
Breton - Nedeleg laouen na bloavezh mat
Bulgarian - Tchestita Koleda; Tchestito Rojdestvo Hristovo
Celtic - Nadolig Llawen a Blwyddyn Newydd Dda
Chinese -
      (Mandarin) Kung His Hsin Nien bing Chu Shen Tan
      (Cantonese) Gun Tso Sun Tan'Gung Haw Sun
      (Hong Kong) Kung Ho Hsin Hsi. Ching Chi Shen Tan
Cornish - Nadelik looan na looan blethen noweth
Cree - Mitho Makosi Kesikansi
Croatian - Sretan Bozic
Czech - Prejeme Vam Vesele Vanoce a stastny Novy Rok
Danish - Gladelig Jul
Dutch - Vrolijk Kerstfeest en een Gelukkig Nieuwjaar
English - Merry Christmas
Esperanto - Gajan Kristnaskon
Estonian - Roomsaid Joulu Puhi
Farsi - Cristmas-e-shoma mobarak bashad
Finnish - Hyvaa joulua
French - Joyeux Noel
Frisian - Noflike Krystdagen en in protte Lok en Seine yn it Nije Jier
German - Froehliche Weihnachten
Greek - Kala Christouyenna
Hawaiian - Mele Kalikimaka
Hebrew - Mo'adim Lesimkha. Chena tova
Hindi - Bada Din Mubarak Ho
Hungarian - Kellemes Karacsonyi unnepeket
Icelandic - Gledileg Jol
Indonesian - Selamat Hari Natal
Iraqi - Idah Saidan Wa Sanah Jadidah
Irish - Nollaig Shona Dhuit
Italian - Buon Natale or Buone Feste Natalizie
Japanese - Shinnen omedeto. Kurisumasu Omedeto
Kala - Khristougena kai Eftikhes to Neon Etos
Korean - Sung Tan Chuk Ha
Latvian - Priecigus Ziemas Svetkus un Laimigu Jauno Gadu
Lettish - Priecigus Ziemassvetkus
Lithuanian - Linksmu Kaledu
Manx - Nollick ghennal as blein vie noa
Maori - Meri Kirihimete
Marathi - Shub Naya Varsh
Navajo - Merry Keshmish
Northern Sotho - Matlhatse le matlhogonolo mo ngwageng o moswa.
Norwegian - God Jul Og Godt Nytt Aar
Pennsylvania German - En frehlicher Grischtdaag un en hallich Nei
Yaahr
Papiamento - Bon Pasku i Felis Anja Nobo
Pig Latin - Errymay ristmaskay
Polish - Wesolych Swiat Bozego Narodzenia
Portuguese - Feliz Natal (Used in Portugal, Brazil and several former Portuguese colonies)
Rapa-Nui - Mata-Ki-Te-Rangi. Te-Pito-O-Te-Henua
Romanian - Craciun Fericit
Russian - Pozdrevlyayu s prazdnikom Rozhdestva i s Novim Godom
Samoan - La Maunia Le Kilisimasi Ma Le Tausaga Fou
Serbian - Hristos se rodi
Scottish - Nollaig Chridheil agus Bliadhna Mhath Ur
Serbian - Hristos se rodi
Singhalese - Subha nath thalak Vewa. Subha Aluth Awrudhak Vewa
Slavonic - Christos Razdajetsja! Slavite Jeho!
Slovak - Sretan Bozic or Vesele vianoce. A stastlivy Novy Rok
Slovene - Vesele Bozicne. Screcno Novo Leto
Spanish - Feliz Navidad (Used in Spain as well as Mexico and most of Central and South America)
Swahili - Krismasi Njema
Swedish - God Jul and (Och) Ett Gott Nytt Ar
Tagalog - Maligayamg Pasko. Masaganang Bagong Taon
Tamil - Nathar Puthu Varuda Valthukkal
Thai - sooK San wan Christmas
Turkish - Noeliniz Ve Yeni Yiliniz Kutlu Olsun
Ukrainian - Srozhdestvom Kristovym
Urdu - Bara Din Mubarak Ho
Vietnamese - Chung Mung Giang Sinh
Waray - Maupay nga Pasko ngan Mainuswagon nga Bag-o nga Tu-ig
Welsh - Nadolig Llawen
Zulu - Nginifisela inhlanhla ne mpumelelo e nyakeni.

Rick Kelly, a Scot/Irish mixed breed and proud 9th generation American

[David Kenny]

Rick,

Very nicely done, as usual.  I just took a quick look at it and your demo worked fine on 32bit Win7 Pro .  I will have to make some time to check it out further.

David

[Rolf Brandt]

Looks very interesting, Rick. I am going to have a closer look at it over the holidays.

Rolf

[Richard Kelly]

Looks very interesting, Rick. I am going to have a closer look at it over the holidays.

Rolf

Let me know your thoughts. The more eyeballs, the better.

I now have SQLite3 dropped in using a connection pool, shared cache memory and WAL mode. Now you know eventually, results have to be passed back. Since this is not going to be some general purpose database server, I plan to always return "records" the client understands. You get results back in a row/column format suitable to PB arrays with UDT records. So now, we need a way to "unroll" to the array in such a way that it can be easily recreated by the client.

Here is preliminary code:

Code Select Expand


%TRUE = 1
%FALSE = 0

TYPE ATest
    iAny AS LONG
    sAny AS ASCIIZ * 14
END TYPE

FUNCTION PBMAIN () AS LONG

LOCAL uRecord       AS ATest
LOCAL sBase         AS STRING
LOCAL nSegmentSize  AS LONG
LOCAL pNextSegment  AS LONG
LOCAL pSegment      AS LONG
LOCAL pReceive      AS LONG
LOCAL nIndex        AS LONG
DIM ar1(1)          AS ATest
DIM ar2(1)          AS ATest

    sBase = ""

    uRecord.iAny = 1
    uRecord.sAny = "abc"

    ar1(0) = uRecord

    uRecord.iAny = 2
    uRecord.sAny = "def"

    ar1(1) = uRecord

    CreateSegment(BYVAL VARPTR(ar1(0)),LEN(uRecord) * ARRAYATTR(ar1(),4),sBase)

    nSegmentSize = 0
    pReceive = VARPTR(ar2(0))
    pSegment = STRPTR(sBase)

    DO WHILE RetrieveSegment(nSegmentSize,pNextSegment,pSegment,pReceive) = %FALSE
    LOOP

    FOR nIndex = 0 TO ARRAYATTR(ar2(),4) - 1

        MSGBOX "Index=" + FORMAT$(nIndex) + ",iAny=" + FORMAT$(ar2(nIndex).iAny) + ",iAny=" + ar2(nIndex).sAny

    NEXT

END FUNCTION
FUNCTION RetrieveSegment(BYREF nSegmentSize AS LONG, _
                         BYREF pNextSegment AS LONG, _
                         BYREF pSegment AS LONG, _
                         BYREF pReceive AS LONG) AS LONG

' Given a segment, parse the elements out
' nSegmentSize should be set 0 on first call
' and not modified between calls.

' pNextSegment is maintained between calls

' Function returns false when there are no more segments to retrieve

' The segment is copied to the pSegment address. Ensure it is large enough to handle the end result

LOCAL pLong     AS LONG PTR

    IF nSegmentSize = 0 THEN

' First 4 bytes of pSegment are the total length

       pLong = pSegment
       nSegmentSize = @pLong
       pNextSegment = pLong + SIZEOF(nSegmentSize)
       pSegment = pSegment + nSegmentSize

    END IF

' Start of segment

    MEMORY COPY pNextSegment, pReceive, nSegmentSize - SIZEOF(nSegmentSize)

    pNextSegment = pNextSegment + nSegmentSize - SIZEOF(nSegmentSize)

    FUNCTION = IIF(pNextSegment >= pSegment,%TRUE,%FALSE)

END FUNCTION
SUB CreateSegment(BYVAL pSegment AS BYTE PTR, _
                  BYVAL nLength AS LONG, _
                  BYREF sBase AS STRING)

' Add length of byte string and byte string to sBase

LOCAL sLength   AS STRING
LOCAL sSegment  AS STRING
LOCAL pLength   AS LONG PTR

    sSegment = ""

' Leave room for the length

    sLength = SPACE$(4)
    pLength = STRPTR(sLength)
    @pLength = nLength

    IF nLength > 0 THEN

       sSegment = SPACE$(nLength)

       MEMORY COPY pSegment, STRPTR(sSegment), nLength

    END IF

    sBase = sLength + sSegment

END SUB


Since PB doesn't allow any dynamic types of data types in UDT's, the memory allocation, at least as I've able to determine, is contiguous for the entire array and the code above can quickly unroll and rebuild the array. It's basically one memory copy most of the time.

In the code above, this is what is built (dumped with a quickie memory dumper I wrote and use quite a lot:

Memory Address=001EE834 Length=40

001EE834  0000  24 00 00 00 01 00 00 00 61 62 63 00 00 00 00 00   $.......abc.....
001EE844  0010  00 00 00 00 00 00 02 00 00 00 64 65 66 00 00 00   ..........def...
001EE854  0020  00 00 00 00 00 00 00 00

Lots of testing ahead....

Rick

[Richard Kelly]

Whew! I now have SQLite3 dropped into the server. I'm testing various configurations to see how options can change busy/locks and when I'm satisified I'll drop it in here and then I'm off to the races as I begin to ramp up my client to use the server. The only basic change I'll make is to add in compression and that will be done only at certain times and it will be the last feature added.

I removed the sqlite3.dll from the release folder and sqlite3.exe from the DB folder under the release folder to get the size down. You'll have to get the latest version at:

http://www.sqlite.org/download.html

In the DB folder run the BAT file to recreate the ctarget.db3 database.

Rick