Concept

The concept behind the whole thing is pretty easy. Take a general purpose PC with a video card in it, and tune it into the general-issue US government channel, Cspan. Next, start a stream rip of the CC stream.
Parse and tag the CC stream and load the data into a MySQL (or other, if there's a better choice out there) database. Finally, allow for full text searching and even email alerting on particular chunks of text as they pop up. So, if the House or Senate happen to go blithering off on something you happen to be concerned about (I.E. health care, taxes, etc), then you will get a transcript for that chunk of time to review as needed.

History

I recall something quite a bit like that a number of years ago on Slashdot, however, several searches have turned up nothing actionable. Since that particular project went under (from what I recall), I decided to bring it back in my own way by seeing how far I would be able to get on my own.
While there are companies which provide this service for a fee, there's no way that I would bring myself to pay for something that is as close as a video capture card and some clever coding.

Tools

The only real tool that I'm aware of would fit the bill would be CCExtractor. While I like the idea of coding up my own stream extractor in Python or some other language, all the hard work has already been done by this clever fellow. Gluing the extractor together with parsing and other information extracting capabilities would be entertaining enough as it is.

Design Requirements

• Capture CC information in 5 minute segments, strip and parse the data so that it is flat text.
• Inject the relevant text into a database along with link to video footage.
• Implement full text search engine on the captured information.
• Write up search and alerting capabilities to the information, including links to the relevant text as well as video footage.

Seems like an interesting project to me. My one and only PC is still downstairs at the moment, acting as a DVR to record my shows. Once I get a new TiVO on order and shipped in, then I will be able to swap it out and begin the project in earnest.

So, any input? Heard of any project quite like this before?

[update]
I did run across http://metavid.org while writing this bit. It's possible that this will do exactly what I want. Still kicking things down and researching.

Cheers,

tom

The more interesting bit about the game is that all the boards that you play are online, created by other users whom have a copy of Atmosphir on their systems.

Some are of pretty good quality, others are simple “run across a few (virtual) miles of power ups and jump really high to his the finish mark”. At the moment, there seems to be a lot more of the latter than the former. However, with new members in the Beta test being added, the level of quality will start coming up eventually.

Along with merely playing the game, you have a full-blown editor in-game (which reminds me a lot of the editor in Sauerbraten) so that you can create your own masterpiece- Or, edit any of the currently existing levels online and save them locally or even upload them as your own.

Installation of the software on the Mac is quick and painless. I like that the developer opted to go with the all-in-one bundle instead of a proper installer application. This will allow for trivial moving, and even deletion of the application without leaving a bunch of libraries scattered all over the place. Unsure of the Windows version, but I assume that it has an installer somewhere.

Preferences are pretty basic at the moment- Movement controls and screen size are about it.

Overall, the performance of the application is pretty good for beta software. There are some hang-ups in the UI whenever the maps are downloaded, and when updates are installed. Overall, for what it offers, and the possibilities afforded, I think this is an acceptable compromise.

So, if interested, check out the Atmosphir web site. There's another review online [Posted Here] that should give you an even better overview than this one has.

Enjoy!

tom

Click on “Read More” for the particulars.

At point A, all of the client information is concatenated together and is hashed together in a one way function (in this instance, I picked sha512, which should be fine as an example). Once that information is hashed together and converted to base64 encoding, the hash is cryptographically signed by a merchant's public key and transmitted.

On the receiving end, the signature is validated. If OK, then the base64 is decoded back and a lookup is performed on the hash information itself at the credit card company's end. Response is sent back to the merchant.

I'm sure there's a lot I'm not thinking of, but it's getting late for me (was up until 0600 this morning mucking with a MythTV setup) and I think it's better to get it out there before I forget everything.

I've also written up a script that will inject 10 million base64 hashes into a MySQL database. It was only after I started down the road on this thought exercise that I was unsure of the latency of trying to perform a SQL select on 173 character strings on a database might cause. More on that later once my generation script finishes.

It's also another reason for me to mess around with Python some more ;)

more later.

tom

import hashlib, string, base64

# Define all our standard variables here.
_first = “jim”
_last = “jones”
_address = “1313 Mockingbird Lane”
_city = “Kansas City”
_state = “Missouri”
_zip = “64119″
_country = “usa”
_ccard = “6011601160116611″
_expiration = “100909″
_cvv = “0210″

# concatenate all input strings
_input = _first + _last + _address + _city + _state + _zip + _country + _ccard + _expiration + _cvv

# take the above and remove all spaces and switch to lower case.
_input1 = str(_input).replace(” “,”").lower()

# create a new hash for the above and then encode to base64 for transmission
print base64.b64encode(hashlib.sha512(_input1).hexdigest())

# Program output is something like (all on one line):
# ODVjYmYyMmQzYTc4NjhhNzk5ODdlNGQxMzAwMTc1ZjYyY2E3NjljYmJiOTZiYjNkNGI1YzRlNmQyMDhkMDViZGNh
# MGFhZjkwYzc5YzRkNDQwZmYyNGI0OTQ0NDdhODVhODE5MTA5ZTc3YzcwNGFiNjY0N2RmMDM5YTA2ODRmMWU=

A friend of mine once said that his typical experience with network security professionals was that if he had hired them to secure a grocery store, they would proceed to install barbed wire fences, attack dogs, searchlights, metal detectors, and perhaps a helicopter or two before finally saying, 'Yep, no one will be shoplifting from here now!' And not only would they have discouraged any sort of legitimate customer from buying from the store, they'd completely ignore the possibility of a teenager with a forged ID buying beer.

Manners, politeness, and respect for fellow human beings as a whole seems to be something that have fallen by the wayside in popular culture. Being rude has been developed into something approaching an art form – an entire genre of comedy is devoted to just this facet of our society.

Combine this with the fact that we, as human beings, are no longer instinctively equipped to judge risks, and you arrive at the modern airline terminal – please take off your shoes and throw away your water, because out of the two billion airline travelers a year, we know you could be one of the few hundred terrorists.

Security isn't a destination, but a process. It's an ideal that will never be attained, not a product packaged into a box. And while there are always tradeoffs, there's no reason we can't combine good security with good customer service.

After some futzing with a fire wire cable, and hitting the “migrate from my old mac” bit during setup, I ended up with pretty much everything I had before, except faster. The new mac setup was probably the single easiest computer migration that I've ever had in my life. All my links, programs, tweaks, and everything in between has ended up on the new box with no observed issues. I think the most strenuous thing I ended up doing was pointing the new mac to the wireless network- since wireless is built-in now.

Since I just run benchmarks with FileScanner on the old mac just last night, I decided to give things a shot on the new computer to see where things were at. According to things, the scan took a hair under 5 minutes to complete against around 18,000 files.

real 4m57.133s
user 0m39.023s
sys 0m30.061s

sqlite> select count(*) from system;
18676

So, according to my guesstimates,

Old Computer:
20063 entries / 1274 seconds == 15.74 entries made per second.

New Computer:
18676 entries / 297 seconds == 62.88 entries made per second.

So, I think I ended up pretty much quadrupled my script performance overnight.

Yeah, I think it's a keeper.

Cheers,

tom

30 years later and there’s still nothing quite like the buddy of my dreams. Of course, now that I’m thinking about it, I’d really like for it to read all of my emails, keep track of my surfing, grab snippets of stuff that ends up on my computer and puts it into an infinitely variable, intelligent soup.

We’re not there yet. Hell, we peek and poke through shitty email interfaces and use up post-it notes in vain attempts to keep track of the infinite torrent of information that visits us on a daily basis. And we still get lost in the middle of a million things, just wanting to go back to being a kid again when the most important thing in the world to you was getting home for dinner before it was dark.

Stopping Things

When I take a look at a chunk of text, I try to imagine what a computer needs to perform. The first step is to attempt to whittle the amount of information which a system needs to pay attention to. Enter stopwords.

Stopwords are exactly what you think they are. Words which the system can toss away and cheerfully ignore. Words like “a”, “and”, “the”, and so on make semantic sense for a human, but are perfectly useless to a computer. Let’s kick around an example a bit to illustrate what a stopword would do for a computer.

We read:
“The quick brown fox jumps over the lazy dog”

A computer reads:
“quick brown fox jumps over lazy dog”

While the “syntactic sugar” of the English language contains a lot of extra fluff to make it more palatable for humans to communicate, quite a bit of it can be discarded and the gist of the communication is still perfectly valid.

Well, that’s the first part of things. Stay tuned for more as it comes.

tom

This is a complex matter, so instead of griping about the dearth of information on what really happens, I'll try my best to document things as I see them.

I'm splitting things up in two sections. REAL and ANALOGY. If the first bit confuses, try the analogy section and see if that helps any. If you have any input on this article or need more information, please reply to this thread over at KCT, or log in with your OpenID account and reply.

First, we'll need a primer on the TCP/IP connection to make the following somewhat more clear to the layperson.

TCP/IP CONNECTIVITY PRIMER

-REAL-
Each IP address on the Internet contains 65535 ports which to access information from. The more popular ports which to send and receive information will be port 80 (HTTP) and port 443 (HTTPS). However, there's still a lot of ports out there in which applications can communicate to each other across the net. Bit Torrent (hereafter BT) binds to ports 6881-6999 and uses them exclusively as transport and communication. This is TCP communication and not UDP communication.

For one system to communicate with another, it requires a specific set of flow control to happen from the initiating system to the server. Assume the one on the left is our client trying to connect to a BT server

[SYN] –>
<–[SYN/ACK]
<–[Communication happens]
[Communication happens]–>
[FIN] –>
<–[FIN]

There's a number of TCP/IP Packet primitives, but here's the ones that I'll be using:

SYN – SYNchronise Packet
ACK – ACKnowledge packet
FIN – No more data from sender
RST – Reset the connection

-ANALOGY-
Take two companies that are trying to communicate to each other via telephone. Both companies have main telephone numbers with a nearly unlimited number of extensions which you can talk to anyone. Normally, x80 and x443 are the most popular, since those both go to the help desk. Mr. Bit Torrent is a busy man, so he has extensions 6881 all the way through to 6999.

For people to talk on the phone, there must me a standard method of communication. Assume that the one of the left is someone who is trying to talk to Mr. Bit Torrent about a business deal

[HELLO?]–>
<–[HELLO, I AM MR BIT TORRENT! THIS IS WHAT I DO ALL DAY (blah blah blah)]
<–[Communication happens]
[Communication happens]–>
[OK, I HAVE TO LEAVE, GOODBYE!] –>
<–[GOODBYE!]

So, that's a normal communication flow in a standard communication that happens trillions of times a day. Now, let's take a look at a (very legal) blocking scenario which Comcast could have performed, but did not.

LEGAL BLOCKING

Blocking two ports can be performed via a firewall, or at the router level. If you do not allow those ports, then no information will be sent via the upstream networking equipment.

-REAL-
[SYN] –> |FIREWALLED| (nothing gets out)
The packets get dropped to the floor and the standard TCP/IP handshake fails. There are methods of circumventing this method- namely by using different TCP/IP ports altogether.

-ANALOGY-
This is easy, call Mr. Torrent's extension and you get a busy signal. Try all of his known extension and get the same busy signal. The only way to make this work otherwise is call a different extension and hope that Mr. Torrent happens to pick up. Since there are 65535 extensions, good luck!

COMCAST (ILLEGAL) BLOCKING

Comcast has went with a different methodology in which they actively break TCP/IP communications via injecting a errant [RST] packet to both the sending and receiving addresses. This is called a MITM (Man-In-The-Middle) Attack. This particular attack is completely unneeded and puts them in liability position since Comcast is masquerading as both ends of the connection and resetting the connection altogether.

-REAL-
[SYN] –>
<–[SYN/ACK]
<–[(Comcast) RST]
[(Comcast) RST]–>
(Connection hits the floor on both sides)

-ANALOGY-
[HELLO?]–>
<–[HELLO, I AM MR BIT TORRENT!]
<–[I'M TERRIBLY BUSY AND CANT TALK ANYMORE! (using Mr. Torrent's voice)]
[I'M TERRIBLY BUSY AND CANT TALK ANYMORE! (using your voice)]–>
(everyone hangs up)

As you see, Comcast is *spoofing* the communication between systems, 'lying' to the other system, and then forcing them to drop the connection. Again, if the relevant ports were blocked, then there would be no liability issue. However, since Comcast makes a point of entering into the conversation and posing as each party, this is fraud and quite illegal.

If Comcast were simply prioritizing packets, that would be one thing. However, the contention is they are spoofing packets back to the clients. Think of it this way, you type in a web address and get back an error message saying the host wasn't available and that error was being generated *by the carrier*, and not the actual website. In that case, the carrier is impersonating the destination and returning false information.

Comcast claims they are not doing this, although some critics have claimed they have irrefutable proof that they are in fact doing that.

Hope this helps.

tom

QnA

What I don't understand is how they can tell what your downloading. Or they can't? They just mess with you if you're downloading anything at all? And who the hell are THEY to police what we do with our (often expensive) cable service?

Since there were reports of people using Lotus notes having problems as well, I can only assume that they are not detecting packets individually (that's something called SPI – Stateful Packet Inspection), but rather just resetting connections if they happen to come across those high ports instead. Using SPI is expensive in terms of time and processing power, whereas just blanket resetting of high port connections is pretty trivial to accomplish.

So basically, if anything hits ports in those regions, Comcast will send a RST to the connection, breaking it. I must strongly reiterate that them performing this is still defrauding users and opening themselves up to liability issues. Block it or don't. But don't play games with the TCP/IP stack in the process.

Links

* http://blog.wired.com/monkeybites/2007/08/its-comcastic-i.html