From hook_l at pyrogon.com  Tue Jan 20 10:46:21 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Tue, 20 Jan 2004 10:46:21 -0500
Subject: OT: Lua questions
In-Reply-To: <20031229022127.GA30993@localhost>
Message-ID: <2004120104621.853773@GATEWAY>

I'm using Lua (I think) in an unrelated project, and was wondering if 
y'all could provide some advice.

I have a database of users managed through SQL (note: I refuse to use 
LuaSQL since it's very poorly documented).  I have a C layer, the Lua 
layer, and then a set of MySQL databases.

I'm trying to figure out the right way to handle basic stuff like 
querying the MySQL db from Lua, etc.

Right now my inclination is that specific queries in Lua are handled 
by proxy through the C layer.  Oh, and I have a telnetd written in C.

So, er, the architecture is kind of like this:

C layer/network driver -> (received text) -> Lua
Lua -> (queries) -> C layer -> (queries) -> SQL db

Using a trivial example, someone telnets to my app and I have to 
verify their authenticity (yes, telnet is all cleartext, not a problem 
for now).  I accept() the connection for now, and then future data 
from that is automatically forwarded to the Lua subsystem, which in 
turn is responsible for prompts, etc. to the user and asking for their 
username, querying the MySQL user database for authenticity, etc.

Any suggestions for the "right" way to handle this?

This will eventually be expanded to handle things like items, etc. 
(this is a MUD, in case that wasn't apparent).

Brian




From nick at rockstarvancouver.com  Wed Jan 21 20:55:20 2004
From: nick at rockstarvancouver.com (Nick Trout)
Date: Wed, 21 Jan 2004 17:55:20 -0800
Subject: [q2x] OT: Lua questions
Message-ID: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC42@iggy.rockstarvancouver.com>


> I'm using Lua (I think) in an unrelated project, and was wondering if
> y'all could provide some advice.
> 
> I have a database of users managed through SQL (note: I refuse to use
> LuaSQL since it's very poorly documented).  I have a C layer, the Lua
> layer, and then a set of MySQL databases.
> 
> I'm trying to figure out the right way to handle basic stuff like
> querying the MySQL db from Lua, etc.
> 
> Right now my inclination is that specific queries in Lua are handled
> by proxy through the C layer.  Oh, and I have a telnetd written in C.
> 
> So, er, the architecture is kind of like this:
> 
> C layer/network driver -> (received text) -> Lua
> Lua -> (queries) -> C layer -> (queries) -> SQL db
> 
> Using a trivial example, someone telnets to my app and I have to
> verify their authenticity (yes, telnet is all cleartext, not a problem
> for now).  I accept() the connection for now, and then future data
> from that is automatically forwarded to the Lua subsystem, which in
> turn is responsible for prompts, etc. to the user and asking for their
> username, querying the MySQL user database for authenticity, etc.
> 
> Any suggestions for the "right" way to handle this?

I suppose that's highly subjective, but personally I'd think about
writing the whole thing in Python. No Lua, No C, and optional MySQL. The
MySQL bit possibly be serialised with Python pickling depending on how
much data you have and the architecture you require. Lua is great for
embedding and extending but you could write you whole system in Python
and avoid the glue/architecture problem. Python is a great language for
prototyping and "freestyle" programming is you're unsure where the
project is going. You'll save a lot of time on refactoring later and
unit testing is a breeze. There are a lot of server side apps written in
Python (e.g. MoinMoin, Zope) which have pretty high performance. Zope
runs off a Python database system called ZODB which you could possibly
use. I wouldn't try using Zope (just ZODB) though as it's really
complicated.

Nick





From hook_l at pyrogon.com  Wed Jan 21 20:59:41 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Wed, 21 Jan 2004 20:59:41 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC42@iggy.rockstarvancouver.com>
Message-ID: <2004121205941.350949@GATEWAY>

> I suppose that's highly subjective, but personally I'd think about
> writing the whole thing in Python. 

Well, shit, you're fired from the Lua Wiki then =P

> require. Lua is great for embedding and extending but you could
> write you whole system in Python and avoid the glue/architecture
> problem. 

True.  Or use Ruby (if it wasn't so slow).  I worry about performance 
though, especially if I have, say, > 1000 simultaneous connected users 
(not a problem right now, but I want to be scalable).

My biggest thing is that I just don't know Python, but I kinda 
half-ass know Lua.  Although I've now had several people say I should 
invert the relationship and just write it all in a scripting language.

When Python/Whatever have a complete integrated development 
environment, I'll maybe bite =)

Actually, I'm curious why there is no Python Studio commercially...

Brian



From nick at rockstarvancouver.com  Wed Jan 21 21:16:33 2004
From: nick at rockstarvancouver.com (Nick Trout)
Date: Wed, 21 Jan 2004 18:16:33 -0800
Subject: [q2x] OT: Lua questions
Message-ID: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC43@iggy.rockstarvancouver.com>




> > I suppose that's highly subjective, but personally I'd think about
> > writing the whole thing in Python.
> 
> Well, shit, you're fired from the Lua Wiki then =P

:)

Look back through the Lua-l archives. I've always stated that Lua is
niche language and excellent for embedding. Python is a PITA to embed.
Python projects are scalable, Lua is awkward to develop on a grand
scale, it's just not mature enough yet. Python has lots of nice language
features Lua doesn't have, but Lua is nice and small and much faster.
Lua is the right tool for game embedding. Python is the right tool for
rapid tool development.
 
> > require. Lua is great for embedding and extending but you could
> > write you whole system in Python and avoid the glue/architecture
> > problem.
> 
> True.  Or use Ruby (if it wasn't so slow).  I worry about performance
> though, especially if I have, say, > 1000 simultaneous connected users
> (not a problem right now, but I want to be scalable).

Sorry, I don't know about Ruby. I've never felt the need for a pure OO
language. Python is a kind of hybrid, like C++. Why I like Python is its
just practical and it works! The libraries are great, docs are great,
the community is great and it's easy to learn.

As far as performance, if you're running it on Ghz machine you should be
able to run that many users. Read about Zope and the performance it
gets. It's entirely written in Python and large corporate website are
written in it; this is content on demand, not static pages.

> My biggest thing is that I just don't know Python, but I kinda
> half-ass know Lua.  Although I've now had several people say I should
> invert the relationship and just write it all in a scripting language.

I think Lua is good for customisation and doing the prompt stuff as you
suggest but then you get into all these architectural issues. I wouldn't
use Python in a game (I had this out at GDC with the Python round
table!) because its not re-entrant, its too big and its too slow. But,
as a standalone tool is above and beyond Lua, and Perl.

> When Python/Whatever have a complete integrated development
> environment, I'll maybe bite =)
> 
> Actually, I'm curious why there is no Python Studio commercially...

There are several solutions, OTTOMH:

 * Activestate (here in Vancouver) provide:
http://www.activestate.com/Products/Visual_Python/

 * There is WingIDE: http://wingide.com/

I don't use any of these, I use, for GUI work (although it contains a
full IDE and target debugger!) Boa:
http://boa-constructor.sourceforge.net/

Or, the PythonWin IDE which comes with win32com. Or the excellent SciTE
editor at http://www.scintilla.org/

BTW for a Python installation definitely use the ActiveState version:
http://www.activestate.com/Products/ActivePython/

It has CHM help and all of the win32 stuff included, along with XML libs
etc.

If you need any help with Python do the Python tutorial as it's
excellent (and I sort of based the Lua one on it) and of course you can
ask questions here.

Nick







From hook_l at pyrogon.com  Wed Jan 21 21:23:17 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Wed, 21 Jan 2004 21:23:17 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC43@iggy.rockstarvancouver.com>
Message-ID: <2004121212317.530991@GATEWAY>

> If you need any help with Python do the Python tutorial as it's
> excellent (and I sort of based the Lua one on it) and of course you
> can ask questions here.

I'm going to give Lua a fair shot for now, and I may quickly change 
directions later.

I'm still highly irate at the lack of good Lua docs.  I've purchased 
Roberto's book, so hopefully it doesn't suck.

Thanks,

Brian




From nick at rockstarvancouver.com  Wed Jan 21 21:26:03 2004
From: nick at rockstarvancouver.com (Nick Trout)
Date: Wed, 21 Jan 2004 18:26:03 -0800
Subject: [q2x] OT: Lua questions
Message-ID: <911F8C8EB7A8084AAEDD55CEDC54D8F84B0BFC@iggy.rockstarvancouver.com>


> I'm still highly irate at the lack of good Lua docs.  

Is the wiki tutorial useful? Are there things missing? I know about the
C API bit missing...

> I've purchased
> Roberto's book, so hopefully it doesn't suck.

I have a copy as well although I haven't got round to reading it yet. I
did read the draft though and it was quite informative. Definitely gives
you insight and ideas I hadn't thought of for Lua.

Nick




From hook_l at pyrogon.com  Wed Jan 21 21:32:29 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Wed, 21 Jan 2004 21:32:29 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <911F8C8EB7A8084AAEDD55CEDC54D8F84B0BFC@iggy.rockstarvancouver.com>
Message-ID: <2004121213229.671228@GATEWAY>

> Is the wiki tutorial useful? Are there things missing? I know about
> the C API bit missing...

Yeah, that happens to be the bit I need =|

Brian

P.S. Do you use IM/ICQ perchance?




From dan at linuxgamepublishing.com  Wed Jan 21 21:54:13 2004
From: dan at linuxgamepublishing.com (Dan Olson)
Date: Wed, 21 Jan 2004 20:54:13 -0600
Subject: [q2x] OT: Lua questions
In-Reply-To: <2004121213229.671228@GATEWAY>
References: <911F8C8EB7A8084AAEDD55CEDC54D8F84B0BFC@iggy.rockstarvancouver.com> <2004121213229.671228@GATEWAY>
Message-ID: <20040122025413.GA26336@localhost>

On Wed, Jan 21, 2004 at 09:32:29PM -0500, Brian Hook wrote:
> > Is the wiki tutorial useful? Are there things missing? I know about
> > the C API bit missing...
> 
> Yeah, that happens to be the bit I need =|
> 

I've done way too much Lua glue code by hand if you have questions.

It wasn't anything too complex and it was Lua 4 but I still may be able
to help.

-- 
Dan Olson


From nicktrout at shaw.ca  Wed Jan 21 23:18:59 2004
From: nicktrout at shaw.ca (Nick Trout)
Date: Wed, 21 Jan 2004 20:18:59 -0800
Subject: [q2x] OT: Lua questions
References: <2004121213229.671228@GATEWAY>
Message-ID: <002201c3e09e$dbc6a870$6400a8c0@quith1>


I did the language syntax partly as docs for our scripters and partly as a
learning exercise for Lua 5. A C API tutorial is pretty low priority, but
its a wiki so other people are free to do it. I'd rather do this quake
project than write a  C API tutorial though :)

I use windows messenger occasionally. I'm nicktrout at hotmail dot com.



----- Original Message ----- 
From: "Brian Hook" <hook_l at pyrogon.com>
To: <q2x at icculus.org>
Sent: Wednesday, January 21, 2004 6:32 PM
Subject: RE: [q2x] OT: Lua questions


> Is the wiki tutorial useful? Are there things missing? I know about
> the C API bit missing...

Yeah, that happens to be the bit I need =|

Brian

P.S. Do you use IM/ICQ perchance?




From hook_l at pyrogon.com  Thu Jan 22 02:20:10 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Thu, 22 Jan 2004 02:20:10 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <002201c3e09e$dbc6a870$6400a8c0@quith1>
Message-ID: <200412222010.823015@GATEWAY>

Oh great and mystical Lua gods, wtf am I doing wrong here?

It's crashing out when I call the update function from C, but if I 
comment out the 'for', it's fine:

clients = {}

function update(cx)
	-- iterate over all active clients and do the right thing
	for a,b in clients do
	end
end

Now, "clients" might be empty, but I thought it would handle that 
correctly, no?  Am I missing something horribly obvious?

Is there any fucking way to get Lua to tell me WHY it's dying either 
during compilation or execution?!

Brian




From hook_l at pyrogon.com  Thu Jan 22 10:56:15 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Thu, 22 Jan 2004 10:56:15 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <200412222010.823015@GATEWAY>
Message-ID: <2004122105615.436849@GATEWAY>

Nevermind...I typed in that code again and it worked.  It's identical 
too...odd.

Brian

On Thu, 22 Jan 2004 02:20:10 -0500, Brian Hook wrote:
> Oh great and mystical Lua gods, wtf am I doing wrong here?
>
>
> It's crashing out when I call the update function from C, but if I
> comment out the 'for', it's fine:
>
> clients = {}
>
>
> function update(cx)
> -- iterate over all active clients and do the right thing for a,b
> in clients do end end
>
> Now, "clients" might be empty, but I thought it would handle that
> correctly, no?  Am I missing something horribly obvious?
>
> Is there any fucking way to get Lua to tell me WHY it's dying
> either during compilation or execution?!
>
> Brian





From nick at rockstarvancouver.com  Thu Jan 22 13:16:43 2004
From: nick at rockstarvancouver.com (Nick Trout)
Date: Thu, 22 Jan 2004 10:16:43 -0800
Subject: [q2x] OT: Lua questions
Message-ID: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC44@iggy.rockstarvancouver.com>




> Oh great and mystical Lua gods, wtf am I doing wrong here?
> 
> It's crashing out when I call the update function from C, but if I
> comment out the 'for', it's fine:
> 
> clients = {}
> 
> function update(cx)
> 	-- iterate over all active clients and do the right thing
> 	for a,b in clients do
> 	end
> end
> 
> Now, "clients" might be empty, but I thought it would handle that
> correctly, no?  Am I missing something horribly obvious?
> 
> Is there any fucking way to get Lua to tell me WHY it's dying either
> during compilation or execution?!

At compile time loadstring/dostring and loadfile/dofile return an error
number.

At runtime you're probably using lua_pcall to call update? E.g.

lua_pushliteral(L, "update");
lua_gettable(L, LUA_GLOBALSINDEX);
// push context?
ret = lua_pcall(L, 1,0, 0);

ret is error value, 0 for none. In the event of a none 0 value there
will be an error message on the stack:

if (ret!=0)
	printf("Error! %s", lua_tostring(L, -1));


Alternatively you can define an error handling function which you insert
as the 3rd argument in pcall.

Nick



From hook_l at pyrogon.com  Thu Jan 22 13:44:13 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Thu, 22 Jan 2004 13:44:13 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC44@iggy.rockstarvancouver.com>
Message-ID: <2004122134413.106953@GATEWAY>

> At compile time loadstring/dostring and loadfile/dofile return an
> error number.

I'm actually calling lua_dobuffer(), in case that makes a difference.

> At runtime you're probably using lua_pcall to call update? E.g.

Actually I was using lua_call, but I'll switch to pcall now, thanks!

Brian




From nick at rockstarvancouver.com  Thu Jan 22 14:51:42 2004
From: nick at rockstarvancouver.com (Nick Trout)
Date: Thu, 22 Jan 2004 11:51:42 -0800
Subject: [q2x] OT: Lua questions
Message-ID: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC46@iggy.rockstarvancouver.com>





> > At runtime you're probably using lua_pcall to call update? E.g.
> 
> Actually I was using lua_call, but I'll switch to pcall now, thanks!

You can also get a full call stack using the debug library when you do
get an error... Bah, that isn't in the wiki tutorial yet!



From hook_l at pyrogon.com  Thu Jan 22 14:53:27 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Thu, 22 Jan 2004 14:53:27 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC46@iggy.rockstarvancouver.com>
Message-ID: <2004122145327.216309@GATEWAY>

> do get an error... Bah, that isn't in the wiki tutorial yet!

The more I try to use Lua, the more pissed I get at the documentation 
=( It's like Luiz and Roberto et. al. just don't give a shit about 
encouraging people to use their stuff, it drives me up a fucking wall.

It would take someone maybe a full day to write a decent introduction 
to Lua that didn't get side tracked into too many details, yet they 
haven't taken the time to do it, and I'm sorry, relying on the 
community for this is just wrong.

cf. my article at www.bookofhook.com on "Tips for Open Source 
Developers" for more =)

Brian






From nick at rockstarvancouver.com  Thu Jan 22 15:01:55 2004
From: nick at rockstarvancouver.com (Nick Trout)
Date: Thu, 22 Jan 2004 12:01:55 -0800
Subject: [q2x] OT: Lua questions
Message-ID: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC47@iggy.rockstarvancouver.com>



> -----Original Message-----
> From: Brian Hook [mailto:hook_l at pyrogon.com]
> Sent: Thursday, January 22, 2004 11:53 AM
> To: q2x at icculus.org
> Subject: RE: [q2x] OT: Lua questions
> 
> > do get an error... Bah, that isn't in the wiki tutorial yet!
> 
> The more I try to use Lua, the more pissed I get at the documentation
> =( It's like Luiz and Roberto et. al. just don't give a shit about
> encouraging people to use their stuff, it drives me up a fucking wall.
> 
> It would take someone maybe a full day to write a decent introduction
> to Lua that didn't get side tracked into too many details, yet they
> haven't taken the time to do it, and I'm sorry, relying on the
> community for this is just wrong.
> 
> cf. my article at www.bookofhook.com on "Tips for Open Source
> Developers" for more =)

I did read that :)

Have you read the papers on Lua? They are quite informative, and the
book has taken Roberto some time and effort, so I can't wholly agree
that there are no docs and they have spent no time on docs.

The docs for Lua are very concise and excellent. There is no bloat,
which makes it very quick to find anything. There are only 3 of them (2
really) maintaining Lua, and Lua is not a full time project (I think) so
they dedicate all their time to improving the language. I have learnt
most of what I know through experimenting (http://doris.sf.net/) and the
lua-l list.

Where I do agree with you is that there is a serious lack of docs for
the C API. I will try hard to find some time to do this but I'm very
busy. I do agree that poor documentation can degrade otherwise excellent
work. One of the first things I look for in a software project is decent
documentation also. AFAIK my wiki docs are the most complete syntax
introduction. All the other docs I've seen are partial or lacking in
information.

Nick




From hook_l at pyrogon.com  Thu Jan 22 15:44:24 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Thu, 22 Jan 2004 15:44:24 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC47@iggy.rockstarvancouver.com>
Message-ID: <2004122154424.876476@GATEWAY>

> Have you read the papers on Lua? 

Which ones, the 7 year old DDJ papers?  Or the four year old random 
symposium presentations? =P

> agree that there are no docs and they have spent no time on docs.

The book is, granted, a monumental effort, but it's taken them to 
revision five to get it out.  

I'm not saying they are obligated to provide documentation, but 
adoption right now is at a much lower rate than it would be simply 
because all help has to be done through lua-l.

And the problem with that is that lua-l is now mostly experts like 
you, Thatcher, etc. and so when I get on there, I feel like a fucking 
dumbass asking questions.

> The docs for Lua are very concise and excellent. 

haha.  "Concise", is that we call it now? =)

The reference manual is woefully lacking in documentation on even the 
most basic aspects of stuff.  It took me forever how to do table 
iteration, and most of it was just guessing on my part because huge 
gaps are there.

> excellent work. One of the first things I look for in a software
> project is decent documentation also. 

To me it ALL starts with the documentation, because that shows me how 
interested the developers are in getting other people on board.  I've 
seen open source projects that effectively just say "read the headers, 
if you're good, you'll figure it out".  And they can kiss my ass =)

The complete lack of comments in the Lua source code sure as shit 
doesn't help their cause much either.

> AFAIK my wiki docs are the
> most complete syntax introduction. 

Yes, they are, and you've done a great job, but I feel bad that the 
onus has fallen on you for this.  can you imagine what the situation 
would be like if you weren't doing that?

Brian




From nick at rockstarvancouver.com  Thu Jan 22 17:57:36 2004
From: nick at rockstarvancouver.com (Nick Trout)
Date: Thu, 22 Jan 2004 14:57:36 -0800
Subject: [q2x] OT: Lua questions
Message-ID: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC48@iggy.rockstarvancouver.com>


> > Have you read the papers on Lua?
> 
> Which ones, the 7 year old DDJ papers?  Or the four year old random
> symposium presentations? =P

:) I suppose they are a bit dated now, but they do give an idea of what
Lua is about. Really, one should be written about Lua 5. There have been
a significant number of changes and Lua is now regarded as a standalone
language, which I don't think it was back then.

> I'm not saying they are obligated to provide documentation, but
> adoption right now is at a much lower rate than it would be simply
> because all help has to be done through lua-l.

I guess they are a little understaffed for full docs. I don't know how
much English being their second language affects this (excellent as
their English is). It may just be time. The few pages I have done took
ages. Docs do seem like a luxury on a lot of projects, although I agree
with you about their usefulness.

> And the problem with that is that lua-l is now mostly experts like
> you, Thatcher, etc. and so when I get on there, I feel like a fucking
> dumbass asking questions.

Brian, noone would call you a dumbass to your face ;-D. And anyway you
could wipe the floor with me with graphics and networking knowledge! I
feel the same way on gdalgorithms :). 

> > The docs for Lua are very concise and excellent.
> 
> haha.  "Concise", is that we call it now? =)

Terse then :)
 
> The reference manual is woefully lacking in documentation on even the
> most basic aspects of stuff.  It took me forever how to do table
> iteration, and most of it was just guessing on my part because huge
> gaps are there.

I guess it's just a reference to me. I don't really go to the reference
manual to learn much stuff. One of the reasons that motivated me to do
the tutorial was to silence constant repetitive questions on lua-l.
Newbie questions do seem to have died down so maybe the tutorial has had
an effect. The lua-users project does seem to get a significant number
of hits, I don't know if the wiki is included in this.

> > excellent work. One of the first things I look for in a software
> > project is decent documentation also.
> 
> To me it ALL starts with the documentation, because that shows me how
> interested the developers are in getting other people on board.  I've
> seen open source projects that effectively just say "read the headers,
> if you're good, you'll figure it out".  And they can kiss my ass =)

:D he he.

Yep, assume makes an ass of you and me.

> The complete lack of comments in the Lua source code sure as shit
> doesn't help their cause much either.

It is a bit cryptic I must admit. The main saving grace is there isn't
that much of it! One of the reasons that I have stuck with Lua is the
infrequency with which I have had to look at the source. It's really
very stable, and since 5 you can do all your customisation with the
vanilla build.

> > AFAIK my wiki docs are the
> > most complete syntax introduction.
> 
> Yes, they are, and you've done a great job, but I feel bad that the
> onus has fallen on you for this.  can you imagine what the situation
> would be like if you weren't doing that?

Thanks very much. It's been quite an enjoyable learning experience doing
it. I'd have thought more people might have put an oar in. I don't know
if they feel intimidated because they feel like it's "my property". I
certainly don't want people going in wholesale and rewriting it, but a
few more contribs would have been nice. Perhaps its because wiki is a
new concept to them.

So here's a question, because I'm not 100% myself: how should the C API
docs be presented? I was thinking of a section introducing binding,
because I think it's an alien concept to most programmers coming into
scripting. Then, how to set up a Lua environment, insert variables,
pushing and pulling data, etc. And, possibly section on different
strategies for embedding and building a project.

Nick




From hook_l at pyrogon.com  Thu Jan 22 19:41:38 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Thu, 22 Jan 2004 19:41:38 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC48@iggy.rockstarvancouver.com>
Message-ID: <2004122194138.568372@GATEWAY>

> :) I suppose they are a bit dated now, but they do give an idea of
> what Lua is about. Really, one should be written about Lua 5. 

The language has changed enough that a lot of the interesting papers, 
notes, etc. are either misleading or just incorrect.

> Brian, noone would call you a dumbass to your face ;-D. 

Right, to my face =)

> I guess it's just a reference to me. I don't really go to the
> reference manual to learn much stuff. One of the reasons that
> motivated me to do the tutorial was to silence constant repetitive
> questions on lua-l.

Right-o, the kinds of questions I think I have asked =)

> It's really very stable, and since 5 you can do all your
> customisation with the vanilla build.

One thing I'd like to see is an op-code limiter so that you can avoid 
runaway/frozen scripts.  I'd retrofit that, but the code is bad enough 
that I don't feel like looking at it that closely =)

> Thanks very much. It's been quite an enjoyable learning experience
> doing it. I'd have thought more people might have put an oar in. I
> don't know if they feel intimidated because they feel like it's "my
> property". I certainly don't want people going in wholesale and
> rewriting it, but a few more contribs would have been nice. Perhaps
> its because wiki is a new concept to them.

Wiki is definitely a new thing for a lot of people, but you should 
probably go ahead and ask people for contribs?

> So here's a question, because I'm not 100% myself: how should the C
> API docs be presented? I was thinking of a section introducing
> binding, because I think it's an alien concept to most programmers
> coming into scripting. Then, how to set up a Lua environment,
> insert variables, pushing and pulling data, etc. And, possibly
> section on different strategies for embedding and building a
> project.

I'm a big believer in incremental learning.  Psychological studies 
have shown that humans don't like rote memorization, and they also 
have a hard time learning lots of things at once without context.

If I were structuring a tutorial, I would do it in a lot of small 
steps.  I'd also not use that interactive environment, since it 
obfuscates the code IMO.

So, for a C API:

Tutorial 1: What is binding, with an example
Tutorial 2: Lua calling C, example w/o args or return values
Tutorial 3: Lua calling C, w/ argos, no return values
Tutorial 4: Lua calling C, w/ args, w/ return values
Tutorial 5: C calling Lua, w/o args or return values
etc. etc.

Small snippets explaining exactly where everything is.  One of the 
hardest concepts to grasp is stack entry placement.  If people would 
just put freakin' comments whenever they access the stack so we have 
an idea of what the stack is supposed to look like at a given point, 
that would be awesome.

Okay, an example of crappy docs: the C API example in the reference 
manual.  They choose a completely contrived, overly complex example, 
such that you have no idea what's going on.  If they just used several 
smaller examples, I think a lot of people would "get it".

Brian




From hook_l at pyrogon.com  Fri Jan 23 23:33:53 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Fri, 23 Jan 2004 23:33:53 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <2004122194138.568372@GATEWAY>
Message-ID: <2004123233353.629361@GATEWAY>

Okay, I sort of have Lua working kinda/sorta like how I expect it to.  
Whew.  So a couple more random questions:

1.  In terms of "making things object oriented", I've seen at least 
two different techniques.  One effectively copies from a global member 
into a new table.

classdef = {}

function classdef:create()
   local new = {}
   for i, v in classdef do
      new[ i ] = v[ i ]
   end
   return new
end

Or something like that.  Then there's the other mechanism that 
involves copying the metatable __index.  Is there a particular 
advantage of one over the other?

2.  This is a bit more complicated, but I wonder what the effect of 
running a script multiple times is?  Specifically, if I have a 
monster.lua, I load it, run it, then later I edit it and reload it, is 
there expected behaviour?

Yeah yeah, I should go join lua-l again...

Brian




From nicktrout at shaw.ca  Sat Jan 24 01:55:08 2004
From: nicktrout at shaw.ca (Nick Trout)
Date: Fri, 23 Jan 2004 22:55:08 -0800
Subject: [q2x] OT: Lua questions
References: <2004123233353.629361@GATEWAY>
Message-ID: <003b01c3e247$012a9870$6400a8c0@quith1>


I think if you want to duplicate the class you should do something like the
following, which just copies all the members.

classdef = {}
function classdef:create()
   local new = {}
   for i, v in classdef do
      new[ i ] = classdef[ i ]
   end
   return new
end

The advantage of the metamethod solution is you only have one instance of
the methods. The instance contains your member variables, but then you have
a table which contains your methods (kind of like they're static). I suppose
it's a bit like a vtable in C++. You can also then overload various
operators, e.g. +, - etc.

Using this method, if you don't find a member you could then look in a super
class if one is attached, i.e. inheritance by delegation. But since this is
a dynamic, runtime operation , if you have to do this a significant number
of times, and there is a significant hierarchy, there could be a speed
penalty. I think the author of Yindo mentioned sometime ago on lua-l that he
would avoid delegation on form a static table of methods as this should be
quicker, i.e. collect all of the methods for a particular inheritance
hierarchy in unique tables so there is only one call.

If you run a file in a state, over a previous run then you'll probably
destroy all the objects that were there, replace them and cause them to be
garbage collected. You may have problems if you have references to any of
these objects from C. It kind of depends on your set up as to what problems
might occur but generally it should just reset.


----- Original Message ----- 
From: "Brian Hook" <hook_l at pyrogon.com>
To: <q2x at icculus.org>
Sent: Friday, January 23, 2004 8:33 PM
Subject: RE: [q2x] OT: Lua questions


Okay, I sort of have Lua working kinda/sorta like how I expect it to.
Whew.  So a couple more random questions:

1.  In terms of "making things object oriented", I've seen at least
two different techniques.  One effectively copies from a global member
into a new table.

classdef = {}

function classdef:create()
   local new = {}
   for i, v in classdef do
      new[ i ] = v[ i ]
   end
   return new
end

Or something like that.  Then there's the other mechanism that
involves copying the metatable __index.  Is there a particular
advantage of one over the other?

2.  This is a bit more complicated, but I wonder what the effect of
running a script multiple times is?  Specifically, if I have a
monster.lua, I load it, run it, then later I edit it and reload it, is
there expected behaviour?

Yeah yeah, I should go join lua-l again...

Brian




From hook_l at pyrogon.com  Sat Jan 24 02:02:38 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Sat, 24 Jan 2004 02:02:38 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <003b01c3e247$012a9870$6400a8c0@quith1>
Message-ID: <20041242238.638619@GATEWAY>

> I think if you want to duplicate the class you should do something
> like the following, which just copies all the members.
>
> classdef = {}
> function classdef:create()
> local new = {}
> for i, v in classdef do
> new[ i ] = classdef[ i ]
> end
> return new
> end

Right, that's what I was going to do, but the Wiki uses the metamethod 
technique, so I was left rather confused.  I'm currently using the 
metamethod technique with some success.

> The advantage of the metamethod solution is you only have one
> instance of the methods. The instance contains your member
> variables, but then you have a table which contains your methods
> (kind of like they're static). I suppose it's a bit like a vtable
> in C++. You can also then overload various operators, e.g. +, - etc.

If I use the metamethod solution, does that mean I can't overload?

c = classdef:create()
c.some_function = derived:somefunction

-- or whatever the syntax would be...

> If you run a file in a state, over a previous run then you'll
> probably destroy all the objects that were there, replace them and
> cause them to be garbage collected. 

That's what I feared. =|

Let's take a simple example.  Say I have a function like:

function object:do_something()
end


It's in its own file, object_do_something.lua.  No other objects are 
defined in there, so I don't have to trash any global variables.  Now, 
I edit and change that function.  The big question is (assuming 
nothing is in the middle of executing): will all previously created 
objects automagically reference the new object:do_something, or will 
they just vomit and die?

Thanks,

Brian




From nicktrout at shaw.ca  Sat Jan 24 13:11:42 2004
From: nicktrout at shaw.ca (Nick Trout)
Date: Sat, 24 Jan 2004 10:11:42 -0800
Subject: [q2x] OT: Lua questions
References: <20041242238.638619@GATEWAY>
Message-ID: <000f01c3e2a5$852b1420$6400a8c0@quith1>

> Right, that's what I was going to do, but the Wiki uses the metamethod
technique, so I was left rather confused.  I'm currently using the
metamethod technique with some success.

I wouldnt use the copy/clone method because it seems inefficient, each
object having a complete copy of its methods. However this technique of
programming is quite popular I think. I think languages that do this are
called prototyped languages, e.g. Apple Newtonscript was quite popular. Not
sure of the implementation details there though.

> > The advantage of the metamethod solution is you only have one
> instance of the methods. The instance contains your member
> variables, but then you have a table which contains your methods
> (kind of like they're static). I suppose it's a bit like a vtable
> in C++. You can also then overload various operators, e.g. +, - etc.

> If I use the metamethod solution, does that mean I can't overload?
c = classdef:create()
c.some_function = derived:somefunction
-- or whatever the syntax would be...

No, you can overload, it just depends how your metamethods are set up. The
usual way is inheritance by delegation. You look at the local method table,
if the member is not there you delegate a parent to look at, thus if you
have the same funtion locally you have overloaded it.

Another way of doing it would be, as described in the last email, by
aggregating all of your methods when the class is instanced. You could check
to see if the inheritance hierarchy you have has been used before e.g.
object : shape : triangle. You could then collect the methods from object,
then shape, then triangle and overload as you go down. These methods could
all live in a single unique method table for that particular hierarchy. If
you had object : shape : box, that could live in another. If you made
another instance of triangle it could used the caches method table to save
duplication. This solution would be faster than delegation since you have no
runtime overhead, just a single table lookup, and you have slight overhead
of one method table per hierarchy.

Lua is wonderfully flexible though, you can pretty much set it up however
appropriate.

> > If you run a file in a state, over a previous run then you'll
> probably destroy all the objects that were there, replace them and
> cause them to be garbage collected.

> That's what I feared. =|
Let's take a simple example.  Say I have a function like:
function object:do_something()
end

> It's in its own file, object_do_something.lua.  No other objects are
defined in there, so I don't have to trash any global variables.  Now,
I edit and change that function.  The big question is (assuming
nothing is in the middle of executing): will all previously created
objects automagically reference the new object:do_something, or will
they just vomit and die?

This is where things could get messy. If you have references to objects, the
references will still reference the old objects. Just think of the
references as pointers to the objects. It's the same thing in C, if you
created a load of new objects, you wouldn't expect your pointers to
magically be directed to the new objects. Whatever you want to happen when
new objects are created is really application specific.

However, if you referenced your functions by string name, rather than
variable name it should work.

The following:

t={}

t.foo = function(x)
        print(x)
    end

foo_ref = t.foo   -- reference formed

foo_ref('hello')

t.foo = function (x)
        print('no!')
    end

foo_ref('there')  -- called using reference
t.foo('there')   -- no ref
t['foo']('there')  -- same as above

-- prints:

hello
there
no!
no!

So, don't make references to objects if you are going to replace the
objects. This is something that you need to be aware of when you are
scripting using garbage collected languages, any language, e.g. Java,
Python, Lua etc. If you use the table to access the object you should be
fine, i.e. t.foo or instance:foo()

Nick



From hook_l at pyrogon.com  Sat Jan 24 17:38:35 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Sat, 24 Jan 2004 17:38:35 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <000f01c3e2a5$852b1420$6400a8c0@quith1>
Message-ID: <2004124173835.314722@GATEWAY>

> technique of programming is quite popular I think. I think
> languages that do this are called prototyped languages, e.g. Apple
> Newtonscript was quite popular. Not sure of the implementation
> details there though.

Yes, this is quite common.  SELF is another language, but JavaScript 
is probably the most prevalent example.

> e.g. Java, Python, Lua etc. If you use the table to access the
> object you should be fine, i.e. t.foo or instance:foo()

Ah, okay, that's what I was planning, so I should be okay.

Thanks!

Brian




From hook_l at pyrogon.com  Wed Jan 28 04:01:18 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Wed, 28 Jan 2004 04:01:18 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <2004124173835.314722@GATEWAY>
Message-ID: <20041284118.702229@GATEWAY>

Okay, my #1 bitch about Lua at this time has become the lack of static 
type/safety/syntax checking.

It really, really sucks having to load and execute a script only to 
find out that you did:

self.name = "whatever"

when you meant:

self.client_name = "whatever"

Or that you did:

obj.function(x,y,z)

when you meant:

obj:function(x,y,z)

Ugh.  It's not writing the scripts that's taking forever, it just the 
real basic syntax checking that's just killing me.

Advice?

Brian




From theoddone33 at icculus.org  Wed Jan 28 13:08:28 2004
From: theoddone33 at icculus.org (theoddone33 at icculus.org)
Date: Wed, 28 Jan 2004 13:08:28 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <20041284118.702229@GATEWAY>
References: <20041284118.702229@GATEWAY>
Message-ID: <1075313308.4017fa9c737fc@mail.icculus.org>

Quoting Brian Hook <hook_l at pyrogon.com>:

> Okay, my #1 bitch about Lua at this time has become the lack of static
> type/safety/syntax checking.

Well, it *is* a dynamically typed language.  I imagine you'd have the same
problem with plenty of other scripting languages.

> Ugh.  It's not writing the scripts that's taking forever, it just the
> real basic syntax checking that's just killing me.

Just like forgetting to end a line with a semicolon oh so many years ago when we
were all learning C, I imagine that this is something that happens less with
more Lua experience.  Other than that, unit tests could speed up the finding of
these errors, and we all should be doing unit tests anyway I'm told.

It's not really something I've run into enough to annoy me.  It's certainly a
downside of dynamic typing, but in my experience these runtime errors are just
about as annoying as compile errors are with C.



From hook_l at pyrogon.com  Wed Jan 28 13:22:03 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Wed, 28 Jan 2004 13:22:03 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <1075313308.4017fa9c737fc@mail.icculus.org>
Message-ID: <200412813223.244084@GATEWAY>

> Well, it *is* a dynamically typed language.  I imagine you'd have
> the same problem with plenty of other scripting languages.

Yes, but in all fairness, you can be dynamically typed but still have 
basic warnings.

> Just like forgetting to end a line with a semicolon oh so many
> years ago when we were all learning C

But you caught that at compile time.

> It's not really something I've run into enough to annoy me.  It's
> certainly a downside of dynamic typing, but in my experience these
> runtime errors are just about as annoying as compile errors are
> with C.

They are WAY more annoying, because they're caught much later in the 
development cycle.  Having something barf after a few minutes of 
run-time is way more disruptive (and dangerous) than something that 
barfs before you startup.

Brian




From nicktrout at shaw.ca  Fri Jan 30 02:58:36 2004
From: nicktrout at shaw.ca (Nick Trout)
Date: Thu, 29 Jan 2004 23:58:36 -0800
Subject: [q2x] OT: Lua questions
References: <200412813223.244084@GATEWAY>
Message-ID: <006601c3e706$dd864490$6400a8c0@quith1>


>> Well, it *is* a dynamically typed language.  I imagine you'd have
>> the same problem with plenty of other scripting languages.
> Yes, but in all fairness, you can be dynamically typed but still have
basic warnings.

It's true Lua could have a few more warnings here and there. Its pretty good
though.

>> It's not really something I've run into enough to annoy me.  It's
> certainly a downside of dynamic typing, but in my experience these
> runtime errors are just about as annoying as compile errors are
> with C.

>They are WAY more annoying, because they're caught much later in the
development cycle.  Having something barf after a few minutes of
run-time is way more disruptive (and dangerous) than something that
barfs before you startup.

It's an unfortunate side effect of dynamic languages, but does mean you can
do more with the objects at runtime and I find productivity much higher
using dynamic languages, because you have to write less code and its more
intuitive to write. You do curse a little hitting your thumb with the hammer
when you're building your framework. But, once its built you can refactor it
into something else much quicker with a dynamic language than a static one.
Prototyping is quicker.

Another unfortunate side effect, and you also get this with dynamic linking
in static languages, is catching extra bloat in your code when you want to
be tight with memory. It's difficult to do static analysis on code to
determine what is necessary.

I haven't used C# but I believe this is some halfway house, as may be Obj-C.
I think C# is some hybrid of Java and Python, i.e. Java with lists and
dictionaries as basic types.So I suppose you can typecheck optional parts of
your code and use variants in other parts? Suppose I ought to take a look at
this but I'm very happy with Python. I still think Lua has a little way to
go with certain features until it becomes a more mature scripting language.

I really dont miss type checking in Python though, and you can always use
its reflexive abilities to typecheck at runtime if necessary, but this
doesn't cure your complaint. Its possible the leap from static programming
to dynamic programming is as great as the one from functional programming to
OO. You just have to be able to read code and spot bugs more readily. Unit
testing is very easy in modular dynamic languages and this may help you too.

Nick




From hook_l at pyrogon.com  Fri Jan 30 03:26:16 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Fri, 30 Jan 2004 03:26:16 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <006601c3e706$dd864490$6400a8c0@quith1>
Message-ID: <200413032616.039749@GATEWAY>

> It's true Lua could have a few more warnings here and there. Its
> pretty good though.

Things I would really like:

1.  The ability to limit VM cycles and force preemption.

2.  At the very least, a compiler flag to warn on assignment to 
uninitialized variables.

3.  A real regex library.  They complain that a POSIX regex 
implementation would take up 4000 lines of code, but at least making 
it optional would have been nice given all the other optional cruft 
they provide.

> It's an unfortunate side effect of dynamic languages, but does mean
> you can do more with the objects at runtime and I find productivity
> much higher using dynamic languages, because you have to write less
> code and its more intuitive to write. 

But you can have both.  Cf. Objective-C, which I still consider the 
finest programming language ever created.  I'd be using it if the 
whole world was running OS X (or NextStep).

Obj-C is fully dynamically typed.  You can send any random message to 
any random object with no problem.  But at the same time, if you do 
provide type information, then it will generate the appropriate 
warning.

> with Python. I still think Lua has a little way to go with certain
> features until it becomes a more mature scripting language.

I appreciate Lua for what it is.  They're trying very hard to keep it 
small and not let it become the be-all end-all language that Python 
and Ruby and other strive for.  They just want something that works 
"Well enough".

So things that we take for granted in other languages, such as 
inheritance or lists or stacks or regex, aren't a part of Lua, and I 
think this is one of the reasons for Lua's success.

Roberto's book, by the way, is pretty good.  Short on examples, but 
otherwise quite good.

> leap from static programming to dynamic programming is as great as
> the one from functional programming to OO. 

Um, not quite, but I understand your point =)

> to read code and spot bugs more readily. Unit testing is very easy
> in modular dynamic languages and this may help you too.

Unit tests are fine, except a lot of the stuff I'm testing can't be 
put into units because they're dependent on my kernel running (if that 
makes sense).  If it was a pure Lua program, that's fine, but since 
it's all embedded, everything is a freakin mess.

Brian





From nick at rockstarvancouver.com  Fri Jan 30 20:20:10 2004
From: nick at rockstarvancouver.com (Nick Trout)
Date: Fri, 30 Jan 2004 17:20:10 -0800
Subject: [q2x] OT: Lua questions
Message-ID: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC63@iggy.rockstarvancouver.com>


> > It's true Lua could have a few more warnings here and there. Its
> > pretty good though.
> 
> Things I would really like:
> 
> 1.  The ability to limit VM cycles and force preemption.

You may be able to do this with coroutines and a line hook. We have a
debugger that can single step.

> 2.  At the very least, a compiler flag to warn on assignment to
> uninitialized variables.

You can do this with metatables.

> 3.  A real regex library.  They complain that a POSIX regex
> implementation would take up 4000 lines of code, but at least making
> it optional would have been nice given all the other optional cruft
> they provide.

There is a posix regex library. Have a look in the Lua addons on the
wiki.

> > leap from static programming to dynamic programming is as great as
> > the one from functional programming to OO.
> 
> Um, not quite, but I understand your point =)

Perhaps not, but I daren't use the phrase "paradigm shift". :)

> > to read code and spot bugs more readily. Unit testing is very easy
> > in modular dynamic languages and this may help you too.
> 
> Unit tests are fine, except a lot of the stuff I'm testing can't be
> put into units because they're dependent on my kernel running (if that
> makes sense).  If it was a pure Lua program, that's fine, but since
> it's all embedded, everything is a freakin mess.

True. Which is part of the reason I'd have engineered the whole thing in
Python. Even if you throw it away and write an optimised version in C,
the (far less painful) prototyping in Python has helped you loads with
where you're going. Its better to extend python than embed it, and you
can unit test. You could extend Lua because it does handle dynamic libs,
then you'd run the unit tests on your modules. It just depends how you
set everything up, ie. Lua has main(), not the exe. You must be able to
run some kinds of tests/diagnostics when the exe fires up?

BTW I've almost got the flyer converted over to Lua in q2x. I've written
a Python script to generate an auto binding wrapper for the q2x stuff.
Even after the refactoring Quake seems quite hardcoded. I think that's
how you envisaged it though Brian, that Lua would just bind to a stub
which is the game DLL. Then the monster scripts and config live in the
game. (Oh, Lua is a DLL as well now, which links with game exe and game
DLL. They all share the same Lua state.) This way you could write
monsters in Lua or C. I think it's going to be too much work for me to
convert all of the monsters over to Lua so I may ask to check it all in
after 2-3 of them and ask for help. i.e. some will be Lua and some C.

Nick






From hook_l at pyrogon.com  Fri Jan 30 20:28:05 2004
From: hook_l at pyrogon.com (Brian Hook)
Date: Fri, 30 Jan 2004 20:28:05 -0500
Subject: [q2x] OT: Lua questions
In-Reply-To: <911F8C8EB7A8084AAEDD55CEDC54D8F851FC63@iggy.rockstarvancouver.com>
Message-ID: <200413020285.566029@GATEWAY>

> You may be able to do this with coroutines and a line hook. We have
> a debugger that can single step.

The various hooks seem to induce a large amount of overhead.  If 
anything, I'd try to do it only during branches (which is how most 
MUDs do it).

> You can do this with metatables.

That's a run time thing, not compile time.

> There is a posix regex library. Have a look in the Lua addons on
> the wiki.

Hmmm, I missed that somehow.  

> BTW I've almost got the flyer converted over to Lua in q2x. 

They not keep you busy up at that studio? =)

> written a Python script to generate an auto binding wrapper for the
> q2x stuff. Even after the refactoring Quake seems quite hardcoded.

It is still very hardcoded, and it's not done being refactored.

> I think that's how you envisaged it though Brian, that Lua would
> just bind to a stub which is the game DLL. Then the monster scripts
> and config live in the game. (Oh, Lua is a DLL as well now, which
> links with game exe and game DLL. They all share the same Lua
> state.) This way you could write monsters in Lua or C. I think it's
> going to be too much work for me to convert all of the monsters
> over to Lua so I may ask to check it all in after 2-3 of them and
> ask for help. i.e. some will be Lua and some C.

I was actually envisioning 100% of the game logic in Lua, like this:

gamex86.dll => gameluax86.dll

Same interface, so the server doesn't notice any change, it's just 
talking to the Lua subsystem.  The Lua subsystem, however, talks to 
all its own Lua modules, and provides entry points for the Lua stuff 
to do heavy-lifting, i.e. kernel traps.

Brian