[cod] Some new cool iptables!
Boyd G. Gafford Ph.D.
drboyd at westportresearch.com
Fri Mar 9 15:30:48 EST 2012
Yeah, that request is a Half-life 2 request, has nothing to do with
Q3-protocol games.
Just make sure the ports you are covering only have Q3-protocol games
running on them.
And 20 should be fine for your LIMITSTAT and LIMITINFO chains.
Thanks,
/ Boyd/
/__________________________________
Boyd G. Gafford Ph.D.
Manager of Software Development
Westport Research Associates Inc.
7001 Blue Ridge Blvd
Raytown, MO 64133
(816) 358-8990
drboyd at westportresearch.com
/
On 03/09/2012 01:42 PM, Paul Gamlowski wrote:
> Yep. I have nowhere near that many per machine. I did the 100 as an
> extreme test. Perhaps another setting is freaking out? :)
>
> EscapedTurkey Billing and Support
> https://escapedturkey.com/helpdesk
>
> On Mar 9, 2012, at 2:38 PM, "Boyd G. Gafford Ph.D."
> <drboyd at westportresearch.com <mailto:drboyd at westportresearch.com>> wrote:
>
>> Yeah, 1000 a second is way too high, that would support like 500
>> games servers on the same hardware lol.
>>
>> We'll take this to a private Email...
>>
>> :)
>>
>> Boyd
>>
>>
>> On 03/09/2012 01:00 PM, escapedturkey wrote:
>>> It's quite a powerful machine, more than 5. =)
>>>
>>> Still doesn't work ..
>>>
>>> /sbin/iptables -N LIMITSTAT
>>> /sbin/iptables -A LIMITSTAT -p udp -m limit --limit 1000/sec
>>> --limit-burst 1000 -j ACCEPT
>>> /sbin/iptables -A LIMITSTAT -p udp -j DROP
>>>
>>> # Add a limit/drop chain for "getinfo" packets that limits it to 10
>>> a second for all servers.
>>> # If you are only protecting one server, you can set the number from
>>> 10 down to 4 (or 2 even).
>>>
>>> /sbin/iptables -N LIMITINFO
>>> /sbin/iptables -A LIMITINFO -p udp -m limit --limit 1000/sec
>>> --limit-burst 1000 -j ACCEPT
>>> /sbin/iptables -A LIMITINFO -p udp -j DROP
>>>
>>> Can you e-mail me so we can experiment?
>>>
>>> Thank you. :)
>>>
>>>
>>> On Fri, Mar 9, 2012 at 1:40 PM, Boyd G. Gafford Ph.D.
>>> <drboyd at westportresearch.com <mailto:drboyd at westportresearch.com>>
>>> wrote:
>>>
>>> How many servers are running on the one machine?
>>>
>>> You might need to up the LIMITSTAT and LIMITINFO chains to more
>>> than 10/sec if you have a bunch of servers being handled by that
>>> rule.
>>>
>>> 10/sec is good for about 4 or 5 game servers.
>>>
>>> You can see if your limits make sense by looking at "sudo
>>> iptables -L -v -n" and looking at dropped packets in the
>>> LIMITSTAT and LIMITINFO chains. If you see some of dropped
>>> packets (but are not under a 'getstatus' or 'getinfo' attack)
>>> then up it from 10 to maybe 20 or so.
>>>
>>>
>>> Thanks,
>>>
>>> /Boyd/
>>> /__________________________________
>>> Boyd G. Gafford Ph.D.
>>> Manager of Software Development
>>> Westport Research Associates Inc.
>>> 7001 Blue Ridge Blvd
>>> Raytown, MO 64133
>>> (816) 358-8990 <tel:%28816%29%20358-8990>
>>> drboyd at westportresearch.com <mailto:drboyd at westportresearch.com>
>>> /
>>>
>>> On 03/09/2012 12:09 PM, escapedturkey wrote:
>>>> Seems to have killed qstat status check.
>>>>
>>>> Without ServerArk:
>>>>
>>>> qstat -old -R -P -a2s 208.43.15.2:27015 <http://208.43.15.2:27015>
>>>> 208.43.15.2:27015 <http://208.43.15.2:27015> "EscapedTurkey.com
>>>> <http://EscapedTurkey.com> Dallas, TX" map de_dust at (null)
>>>> 0/32 players 1 ms
>>>>
>>>> ServerArk running:
>>>>
>>>> qstat -old -R -P -a2s 208.43.15.2:27015 <http://208.43.15.2:27015>
>>>> 208.43.15.2:27015 <http://208.43.15.2:27015> no response
>>>>
>>>> This is what I am using:
>>>>
>>>> # The main logic of ServerArk, all done with /sbin/iptables!
>>>>
>>>> # Version 1.01
>>>> # (C) 2012 Boyd G. Gafford Ph.D. (Usage is under the LGPL)
>>>> # To contact me, simply post on the forum at elitewarriors.net
>>>> <http://elitewarriors.net>.
>>>> #
>>>> # Please note these rules ONLY affect UDP packets to the game
>>>> servers, nothing else!
>>>> # This script will protect all Q3-protocol servers on the port
>>>> 28960. It protects
>>>>
>>>> # against both 'getstatus' and 'getinfo' attacks, as well as
>>>> 'getchallenge' atttacks,
>>>> # even from a UDP flood with random source IPs.
>>>>
>>>> # Add a limit/drop chain for "getstatus" packets that limits it
>>>> to 10 a second for all servers.
>>>> # If you are only protecting one server, you can set the number
>>>> from 10 down to 4 (or 2 even).
>>>>
>>>> /sbin/iptables -N LIMITSTAT
>>>> /sbin/iptables -A LIMITSTAT -p udp -m limit --limit 10/sec
>>>> --limit-burst 10 -j ACCEPT
>>>> /sbin/iptables -A LIMITSTAT -p udp -j DROP
>>>>
>>>>
>>>> # Add a limit/drop chain for "getinfo" packets that limits it
>>>> to 10 a second for all servers.
>>>> # If you are only protecting one server, you can set the number
>>>> from 10 down to 4 (or 2 even).
>>>>
>>>> /sbin/iptables -N LIMITINFO
>>>> /sbin/iptables -A LIMITINFO -p udp -m limit --limit 10/sec
>>>> --limit-burst 10 -j ACCEPT
>>>> /sbin/iptables -A LIMITINFO -p udp -j DROP
>>>>
>>>>
>>>> # Add a limit/drop chain for "getchallenge" packets that limits
>>>> it to 5 a second for all servers.
>>>> # If you are only protecting one server, you can set the number
>>>> from 5 down to 2. Setting it
>>>> # at 2 means only 2 players could connect to the server per
>>>> second. Set LIMITCONN to the
>>>> # same, as there is one getchallenge/connect packet sequence
>>>> per valid player connection.
>>>>
>>>> /sbin/iptables -N LIMITCHLG
>>>> /sbin/iptables -A LIMITCHLG -p udp -m limit --limit 5/sec
>>>> --limit-burst 5 -j ACCEPT
>>>> /sbin/iptables -A LIMITCHLG -p udp -j DROP
>>>>
>>>>
>>>> # Add a limit/drop chain for "connect" packets that limits it
>>>> to 5 a second for all servers.
>>>> # If you are only protecting one server, you can set the number
>>>> from 5 down to 2. Setting it
>>>> # at 2 means only 2 players could connect to the server per
>>>> second. Set LIMITCHLG to the
>>>> # same, as there is one getchallenge/connect packet sequence
>>>> per valid player connection.
>>>>
>>>> /sbin/iptables -N LIMITCONN
>>>> /sbin/iptables -A LIMITCONN -p udp -m limit --limit 5/sec
>>>> --limit-burst 5 -j ACCEPT
>>>> /sbin/iptables -A LIMITCONN -p udp -j DROP
>>>>
>>>>
>>>> # Add a limit chain that prevents more than 70 packets a second
>>>> per player.
>>>> # This is the main logic of ServerArk, but just performed by an
>>>> iptable rule.
>>>> # We allow up to 128 players which is enough for 4 servers full
>>>> (at 32 players each).
>>>> # If you only have one server, you could the size and max to 32.
>>>> # If you have players who have manually set their packet rate
>>>> up to 100, just change the 70 to 100.
>>>>
>>>> /sbin/iptables -N LIMITPLRS
>>>> /sbin/iptables -A LIMITPLRS -p udp -m hashlimit
>>>> --hashlimit-name PLAYERS --hashlimit-above 100/sec
>>>> --hashlimit-burst 100 --hashlimit-mode srcip,srcport
>>>> --hashlimit-htable-size 128 --hashlimit-htable-max 128
>>>> --hashlimit-htable-gcinterval 1000 --hashlimit-htable-expire
>>>> 10000 -j DROP
>>>> /sbin/iptables -A LIMITPLRS -p udp -j ACCEPT
>>>>
>>>>
>>>> # Add the rules to pick out the various special packets and
>>>> send them to appropriate limit chains.
>>>> # To protect 5 ports, just specify a range like "--dport
>>>> 28960:28964" below.
>>>>
>>>> /sbin/iptables -A INPUT -p udp --dport 27000:30000 -m string
>>>> --string "getstatus" --algo bm --from 32 --to 33 -j LIMITSTAT
>>>> /sbin/iptables -A INPUT -p udp --dport 27000:30000 -m string
>>>> --string "getinfo" --algo bm --from 32 --to 33 -j LIMITINFO
>>>> /sbin/iptables -A INPUT -p udp --dport 27000:30000 -m string
>>>> --string "getchallenge" --algo bm --from 32 --to 33 -j LIMITCHLG
>>>> /sbin/iptables -A INPUT -p udp --dport 27000:30000 -m string
>>>> --string "connect" --algo bm --from 32 --to 33 -j LIMITCONN
>>>>
>>>>
>>>> # Send all other packets (normal player packets) to the limit
>>>> players chain.
>>>> # A port range like "--dport 28960:28964" could also be used
>>>> here as well.
>>>> # /sbin/iptables -A INPUT -p udp --dport 28960 -j LIMITPLRS
>>>>
>>>> /sbin/iptables -A INPUT -p udp --dport 27000:30000 -j LIMITPLRS
>>>>
>>>> On Fri, Mar 9, 2012 at 11:41 AM, Boyd G. Gafford Ph.D.
>>>> <drboyd at westportresearch.com
>>>> <mailto:drboyd at westportresearch.com>> wrote:
>>>>
>>>> Yeah, if they flood "getstatus" and "getinfo", during the
>>>> attack your server will not be visible from the master list.
>>>>
>>>> If they flood "getchallenge", during the attack nobody will
>>>> be able to join your server.
>>>>
>>>> Once the attack ends, then you'll be visible again and
>>>> people can join normally.
>>>>
>>>> Since most of these attacks are from spoofed random IP
>>>> addresses (millions of them), you can't limit per IP, as no
>>>> IP repeats.
>>>>
>>>> This set of rules is about the best I've found short of
>>>> doing a whitelisted server, where you only allow IP's of
>>>> known good players, and block everything else, and then
>>>> people have to join the server with "connect IP:PORT".
>>>> That's fairly inconvenient for most players, so these rules
>>>> are about as good as you can get and still allow usage from
>>>> the master list.
>>>>
>>>> Thanks,
>>>>
>>>> / Boyd/
>>>>
>>>> /__________________________________
>>>> Boyd G. Gafford Ph.D.
>>>> Manager of Software Development
>>>> Westport Research Associates Inc.
>>>> 7001 Blue Ridge Blvd
>>>> Raytown, MO 64133
>>>> (816) 358-8990 <tel:%28816%29%20358-8990>
>>>> drboyd at westportresearch.com
>>>> <mailto:drboyd at westportresearch.com>
>>>> /
>>>>
>>>> On 03/09/2012 10:00 AM, Ruediger Meier wrote:
>>>>> On Friday 09 March 2012, Boyd G. Gafford Ph.D. wrote:
>>>>>> Just wanted to share these with the COD group here. I've been
>>>>>> running these rules for about a week now, and they have been working
>>>>>> wonderfully. Let me know if you end up using them and how they work
>>>>>> for you.
>>>>> Be aware that now it's easy for a attacker to make your servers
>>>>> invisible for others by flooding your limit rules.
>>>>> Maybe you should rather limit per ip.
>>>>>
>>>>> cu,
>>>>> Rudi
>>>>>
>>>>>> #!/bin/bash
>>>>>> # The main logic of ServerArk, all done with iptables!
>>>>>> # Version 1.01
>>>>>> # (C) 2012 Boyd G. Gafford Ph.D. (Usage is under the LGPL)
>>>>>> # To contact me, simply post on the forum atelitewarriors.net <http://elitewarriors.net>.
>>>>>> #
>>>>>> # Please note these rules ONLY affect UDP packets to the game
>>>>>> servers, nothing else!
>>>>>> # This script will protect all Q3-protocol servers on the port 28960.
>>>>>> It protects
>>>>>> # against both 'getstatus' and 'getinfo' attacks, as well as
>>>>>> 'getchallenge' atttacks,
>>>>>> # even from a UDP flood with random source IPs.
>>>>>>
>>>>>> # Add a limit/drop chain for "getstatus" packets that limits it to 10
>>>>>> a second for all servers.
>>>>>> # If you are only protecting one server, you can set the number from
>>>>>> 10 down to 4 (or 2 even).
>>>>>> iptables -N LIMITSTAT
>>>>>> iptables -A LIMITSTAT -p udp -m limit --limit 10/sec --limit-burst 10
>>>>>> -j ACCEPT
>>>>>> iptables -A LIMITSTAT -p udp -j DROP
>>>>>>
>>>>>> # Add a limit/drop chain for "getinfo" packets that limits it to 10 a
>>>>>> second for all servers.
>>>>>> # If you are only protecting one server, you can set the number from
>>>>>> 10 down to 4 (or 2 even).
>>>>>> iptables -N LIMITINFO
>>>>>> iptables -A LIMITINFO -p udp -m limit --limit 10/sec --limit-burst 10
>>>>>> -j ACCEPT
>>>>>> iptables -A LIMITINFO -p udp -j DROP
>>>>>>
>>>>>> # Add a limit/drop chain for "getchallenge" packets that limits it to
>>>>>> 5 a second for all servers.
>>>>>> # If you are only protecting one server, you can set the number from
>>>>>> 5 down to 2. Setting it
>>>>>> # at 2 means only 2 players could connect to the server per second.
>>>>>> Set LIMITCONN to the
>>>>>> # same, as there is one getchallenge/connect packet sequence per
>>>>>> valid player connection.
>>>>>> iptables -N LIMITCHLG
>>>>>> iptables -A LIMITCHLG -p udp -m limit --limit 5/sec --limit-burst 5
>>>>>> -j ACCEPT
>>>>>> iptables -A LIMITCHLG -p udp -j DROP
>>>>>>
>>>>>> # Add a limit/drop chain for "connect" packets that limits it to 5 a
>>>>>> second for all servers.
>>>>>> # If you are only protecting one server, you can set the number from
>>>>>> 5 down to 2. Setting it
>>>>>> # at 2 means only 2 players could connect to the server per second.
>>>>>> Set LIMITCHLG to the
>>>>>> # same, as there is one getchallenge/connect packet sequence per
>>>>>> valid player connection.
>>>>>> iptables -N LIMITCONN
>>>>>> iptables -A LIMITCONN -p udp -m limit --limit 5/sec --limit-burst 5
>>>>>> -j ACCEPT
>>>>>> iptables -A LIMITCONN -p udp -j DROP
>>>>>>
>>>>>> # Add a limit chain that prevents more than 70 packets a second per
>>>>>> player. # This is the main logic of ServerArk, but just performed by
>>>>>> an iptable rule.
>>>>>> # We allow up to 128 players which is enough for 4 servers full (at
>>>>>> 32 players each).
>>>>>> # If you only have one server, you could the size and max to 32.
>>>>>> # If you have players who have manually set their packet rate up to
>>>>>> 100, just change the 70 to 100.
>>>>>> iptables -N LIMITPLRS
>>>>>> iptables -A LIMITPLRS -p udp -m hashlimit --hashlimit-name PLAYERS
>>>>>> --hashlimit-above 70/sec --hashlimit-burst 70 --hashlimit-mode
>>>>>> srcip,srcport --hashlimit-htable-size 128 --hashlimit-htable-max 128
>>>>>> --hashlimit-htable-gcinterval 1000 --hashlimit-htable-expire 10000 -j
>>>>>> DROP iptables -A LIMITPLRS -p udp -j ACCEPT
>>>>>>
>>>>>> # Add the rules to pick out the various special packets and send them
>>>>>> to appropriate limit chains.
>>>>>> # To protect 5 ports, just specify a range like "--dport 28960:28964"
>>>>>> below. iptables -A INPUT -p udp --dport 28960-m string --string
>>>>>> "getstatus" --algo bm --from 32 --to 33 -j LIMITSTAT
>>>>>> iptables -A INPUT -p udp --dport 28960-m string --string "getinfo"
>>>>>> --algo bm --from 32 --to 33 -j LIMITINFO
>>>>>> iptables -A INPUT -p udp --dport 28960-m string --string
>>>>>> "getchallenge" --algo bm --from 32 --to 33 -j LIMITCHLG
>>>>>> iptables -A INPUT -p udp --dport 28960-m string --string "connect"
>>>>>> --algo bm --from 32 --to 33 -j LIMITCONN
>>>>>>
>>>>>> # Send all other packets (normal player packets) to the limit players
>>>>>> chain. # A port range like "--dport 28960:28964" could also be used
>>>>>> here as well. iptables -A INPUT -p udp --dport 28960-j LIMITPLRS
>>>>>> /
>>>>>> /Also, you can do an "iptables -L -v -n" to see what kind of attacks
>>>>>> these rules have blocked. Here's an example of this command after a
>>>>>> "getchallenge" flood attack from random IPs, on our Dallas server
>>>>>> running on port 29070.
>>>>>>
>>>>>> ew at server1:~$ sudo iptables -L -v -n
>>>>>> Chain INPUT (policy ACCEPT 11368 packets, 1538K bytes)
>>>>>> pkts bytes target prot opt in out source
>>>>>> destination
>>>>>> 3880 177K LIMITSTAT udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> udp dpt:29070 STRING match "getstatus" ALGO name
>>>>>> bm FROM 32 TO 33
>>>>>> 14036 617K LIMITINFO udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> udp dpt:29070 STRING match "getinfo" ALGO name bm
>>>>>> FROM 32 TO 33
>>>>>> 37M 1620M LIMITCHLG udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> udp dpt:29070 STRING match "getchallenge" ALGO
>>>>>> name bm FROM 32 TO 33
>>>>>> 17 4989 LIMITCONN udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> udp dpt:29070 STRING match "connect" ALGO name bm
>>>>>> FROM 32 TO 33
>>>>>> 237K 17M LIMITPLRS udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> udp dpt:29070
>>>>>>
>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>> pkts bytes target prot opt in out source
>>>>>> destination
>>>>>>
>>>>>> Chain OUTPUT (policy ACCEPT 343K packets, 54M bytes)
>>>>>> pkts bytes target prot opt in out source
>>>>>> destination
>>>>>>
>>>>>> Chain LIMITCHLG (1 references)
>>>>>> pkts bytes target prot opt in out source
>>>>>> destination
>>>>>> 40025 1761K ACCEPT udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> limit: avg 5/sec burst 5
>>>>>> *37M 1618M DROP* udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0>
>>>>>>
>>>>>> Chain LIMITCONN (1 references)
>>>>>> pkts bytes target prot opt in out source
>>>>>> destination
>>>>>> 17 4989 ACCEPT udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> limit: avg 5/sec burst 5
>>>>>> 0 0 DROP udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0>
>>>>>>
>>>>>> Chain LIMITINFO (1 references)
>>>>>> pkts bytes target prot opt in out source
>>>>>> destination
>>>>>> 14036 617K ACCEPT udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> limit: avg 10/sec burst 10
>>>>>> 0 0 DROP udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0>
>>>>>>
>>>>>> Chain LIMITPLRS (1 references)
>>>>>> pkts bytes target prot opt in out source
>>>>>> destination
>>>>>> 1642 104K DROP udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> limit: above 70/sec burst 70 mode srcip-srcport
>>>>>> htable-size 128 htable-max 128
>>>>>> 236K 17M ACCEPT udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0>
>>>>>>
>>>>>> Chain LIMITSTAT (1 references)
>>>>>> pkts bytes target prot opt in out source
>>>>>> destination
>>>>>> 3868 177K ACCEPT udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0> limit: avg 10/sec burst 10
>>>>>> 12 516 DROP udp -- * *0.0.0.0/0 <http://0.0.0.0/0>
>>>>>> 0.0.0.0/0 <http://0.0.0.0/0>
>>>>>>
>>>>>> Notice the bolded packet/byte statistics for the "getchallenge" drop
>>>>>> chain named LIMITCHLG. A total of 37 million packets dropped. I was
>>>>>> on the game during this attack, and although the server did lag a bit
>>>>>> from the sheer size of the flood (almost saturating the bandwidth),
>>>>>> nobody lagged out. Without this rule, the game server deadlocked.
>>>>>>
>>>>>> Also notice you can tell how many players have connected to the
>>>>>> server, as the LIMITCONN status shows 17 packets accepted. So during
>>>>>> this time we had 17 players join the game.
>>>>>>
>>>>>> You can also see how many people requested the servers in game (as
>>>>>> well as other services like GameTracker getting info on you), as that
>>>>>> corresponds to the LIMITSTAT and LIMITINFO chains.
>>>>>>
>>>>>> Another cool thing you can do is "cat /proc/srv/ipt_hashlimit/PLAYERS
>>>>>> to see the IP addresses of all the players currently connected to the
>>>>>> server(s). Once a player quits playing, he goes out of this file
>>>>>> automatically after 10 seconds.
>>>>>>
>>>>>> I may refine these a bit further, but for now, these seem to be
>>>>>> working well on our VPS.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /Boyd/
>>>>> _______________________________________________
>>>>> cod mailing list
>>>>> cod at icculus.org <mailto:cod at icculus.org>
>>>>> http://icculus.org/mailman/listinfo/cod
>>>>>
>>>>
>>>> _______________________________________________
>>>> cod mailing list
>>>> cod at icculus.org <mailto:cod at icculus.org>
>>>> http://icculus.org/mailman/listinfo/cod
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> EscapedTurkey.com <http://EscapedTurkey.com> Billing and Support
>>>> https://www.escapedturkey.com/helpdesk
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> cod mailing list
>>>> cod at icculus.org <mailto:cod at icculus.org>
>>>> http://icculus.org/mailman/listinfo/cod
>>>
>>> _______________________________________________
>>> cod mailing list
>>> cod at icculus.org <mailto:cod at icculus.org>
>>> http://icculus.org/mailman/listinfo/cod
>>>
>>>
>>>
>>>
>>> --
>>> EscapedTurkey.com <http://EscapedTurkey.com> Billing and Support
>>> https://www.escapedturkey.com/helpdesk
>>>
>>>
>>>
>>> _______________________________________________
>>> cod mailing list
>>> cod at icculus.org
>>> http://icculus.org/mailman/listinfo/cod
>> _______________________________________________
>> cod mailing list
>> cod at icculus.org <mailto:cod at icculus.org>
>> http://icculus.org/mailman/listinfo/cod
>
>
> _______________________________________________
> cod mailing list
> cod at icculus.org
> http://icculus.org/mailman/listinfo/cod
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://icculus.org/pipermail/cod/attachments/20120309/bcde84d2/attachment-0001.htm>
More information about the cod
mailing list