Last week, security experts launched a sneak attack to disconnect Troyak, an Internet service provider in Eastern Europe that served as a global gateway to a nest of cyber crime activity. For the past seven days, unnamed members of the security community reportedly have been playing Whac-a-Mole with Troyak, which has bounced from one legitimate ISP to the next in a bid to reconnect to the wider Internet.
But experts say Troyak’s apparent hopscotching is expected behavior from what is in fact a carefully architected, round-robin network of backup and redundant carriers, all designed to keep a massive organized criminal operation online should a disaster like the Troyak disconnection strike.
Security firm RSA believes Troyak is but one of five upstream providers that encircle a nest of eight so-called “bulletproof networks” – Web hosting providers considered impervious to takedown by local law enforcement (pictured in red in the graphic below). RSA said this group of eight hosts some of the Internet’s largest concentrations of malicious software, including password stealing banking Trojans like ZeuS and Gozi, as well as huge repositories of personal and financial data stolen by these Trojans and a notorious Russian phishing operation known as RockPhish.
According to a report RSA issued today, these eight networks connect directly to Troyak and four other upstream providers that “surround the malicious core,” and help to “mask the true malware-hosting armada and provide solid uptime to the malware servers.” In addition, Troyak and the other four upstream providers (shown in orange in the diagram above) all share connections amongst themselves, and individually connect to one or more legitimate, regional ISPs (the green circles in the picture above) that can provide connections to the global Internet.
In fact, RSA said, when Troyak was initially knocked offline on March 9, it was because several regional ISPs (green networks on the left side of the graphic) simultaneously denied it service. Presumably, these ISPs cut the cord to Troyak due to pressure from security researchers who enumerated and explained to those ISPs the criminal networks they were supporting.
The trouble was, the four other providers in Troyak’s hub also had their own connections to regional ISPs, and so the entire network of bulletproof hosts that largely depended on Troyak to reach the larger Internet could suddenly shift gears and connect to the Web through these peers. The regional ISPs are depicted in the green circles in the map above, and RSA calls them legitimate ISPs, although anti-spam outfit Spamhaus on Tuesday listed one of Troyak’s main regional connection — Russia-based NLINE — on its spam blacklist for “repeatedly hosting cybercriminal spam gangs.”
“It is important to understand that although part of this infrastructure may lose connectivity, these bulletproof networks are still able to resume online activity through other upstream providers they have access to; most are back online having accessed alternate connections within that same cybercrime infrastructure,” RSA stated in its report on the Troyak takedown. “This redundancy mechanism is at the core of keeping malicious servers up and running over time, as observed through the past week’s events.”
RSA isn’t alone in trying to map badness and ISP reputation on the Web. In an excellently timed-paper, a trio of university researchers released a study this week at the IEEE Infocom conference in San Diego that used data from at least a dozen spam, malware, bot and phishing blacklists to identify malicious networks. The researchers, from the Oak Ridge National Laboratory and Indiana Unversity at Bloomington, identified several dense clusters of ISPs – particularly in Ukraine and Turkey – that appeared to be overly tolerant of activity emanating from their networks.
For example, the researchers also sought to identify ISPs and hosting providers that had a disproportionate number of network peers that were malicious. For this measurement, they focused on ISPs with at least three such partner networks. They found 22 networks that had 100 percent of their customers classified as malicious, while some 194 networks had at least 50 percent of their customers fall into that category.
A story I wrote on that study can be found in today’s online edition of MIT Technology Review, at this link here.
I’ll be writing more about other data-driven efforts to identify problem ISPs and hosting providers over the next few days. Stay tuned.
Are we getting close to the point where companies (and perhaps entire countries) simply refuse to route traffic to/from Russia and other nations that seem to permit the hosting of these known bad sites?
Before blacklisting Russian sites, wouldn’t it be better to try some diplomatic channels first? Putin and Medvedev may not care much about Westerners losing their money, but they probably care less about the cybercriminals inside their border. In theory, a offensive by Hillary Clinton, David Miliband and other foreign ministers may convey to them the scope of the problem.
Course, you gotta get Western politicians to understand the scope of the problem first. That’s the rub.
The Russian officials are not very interested in controlling these criminals, many researchers and law enforcement officials have indicated they use the bot-nets to squash dissent on opposing political sites. They aren’t ready to get rid of such good censoring as of yet.
Also I suspect they figure they may need it as a military weapon to use on enemies in a nuclear crisis.
It seems DARPA was right in making a bullet proof model of communication to survive a nuclear holocaust – it took the worlds criminals to find a way to make it truly bullet proof.
Let’s hope not. You are talking about blocking entire countries the way Iran and China do. You are talking about creating a planet of division – a planet with no means of full communication.
This is the length you will go to. Instead of getting your operating system act together.
This is what we fear will happen if I understand what many fellow members at my favorite site feel, is that the least interference used to fight the problem the better.
We don’t want a locked down internet bereft of free exchange of ideas. It is this free exchange that gives the free world our leg up. Especially in the economy. I don’t think draconian measures are necessary. Even some mild self policing could help – financial incentives for end users would be very effective, I’d think.
It isn’t like a lot of ISPs have not been pro active, AT&T offers free anti-virus for many of its customers. The only problem is some ISPs make bad selections for the AV solutions they push. However they are better than nothing.
No one is talking about blocking entire countries — instead, why can’t Russia (for example) do internally what many western countries already do, and simply block or prosecute known criminals? Send some Russian law enforcement down to VISHCLUB headquarters, serve them with a warrant, unplug their servers and throw them in jail. I’m sure it’s somehow more complicated than that, but WHY? Corruption?
@Nick: Some U.S. ISPs already block all email from large areas of the world, like Asia and Eastern Europe. But suppose you are an internet provider in Asia, and no matter how proactive you are in keeping your network free of sources of spam, your business customers still can’t send emails to their US clients due to that kind of blocking. What is your motivation to bother keeping your network clean? Now suppose *your* customers’ emails are getting through, but not those from your competitor’s network full of open proxies. Business customers in your country will move their accounts to your network to improve their opportunities for international transactions. The increase in customers is a significant incentive for you to continue your responsible policies.
Some of the members at inboxrevenge.com have been making an effort to report any spamvertised site on the various free hosting providers. Russian hosting providers, like Pochta.ru, actually tend to be the fastest to respond to reports (despite the additional expense it must cost them to have English speaking support staff handling the complaints). Microsoft’s Spaces.Live.com, in contrast, allows the spamvertised blogs to stay alive for weeks or months, even when they are selling pirated copies of Microsoft products!
The point is that there are enlightened, clueless and criminal internet providers in every country. A ham-fisted solution like blocking an entire country doesn’t hurt the criminals much — they’ve got open proxies that can relay their spam through whatever route is necessary to get to the trojan-infected computers in the U.S. they’re going to use to mail it. But it can be devastating for honest internet users.
Thanks for this extremely informative posting. This is right up my alley.
I’d like to extrapolate what percentage of traffic from all of this relates back to the renowned “Canadian Pharmacy” criminal pill sites, operated by Spamit and Glavmed. I’d guess at least 70 – 80%. It’s usually associated very closely with a wide variety of malware out there.
SiL / IKS / concerned citizen
Such reports are always troubling. In the aggregate, they are alarming. Unfortunately, the more that criminals exploit the wide-open, few-rules internet, the more likely it becomes that at least the US government will impose its heavy hand on both providers and users (in order to keep us “safe and secure,” of course). It takes little imagination to project that, sooner than later, using the internet will become as free and convenient an undertaking as flying on a commercial airliner. The technological and statutory wherewithal are already in place to do cyber strip-searches of everyone who uses the net. How long before they become mandatory and routine for all users every time they log on?
Happy St. Patrick’s Day all. Mark, your comment comes very close to my answer to the 7-minute interview I gave to Team Cymru a few weeks ago, which asked me “What Keeps You Up at Night?”
http://www.krebsonsecurity.com/2010/01/what-keeps-you-up-at-night/
It looks like governments may not be necessary…Troyak has just been de-peered again:
http://www.net-security.org/secworld.php?id=9039
This is the kind of self-policing that the Internet needs to survive without government intervention. Once the bad guys are exposed (after losses of millions by the public) it seems that nobody wants to be associated with them. Let’s hope this example stands the test of time.
Thanks for the link to your interview that I had misses, Brian. Great minds…
If you abuse postal regulations grossly enough the Postal Service can deny you mailing privileges. Can’t ICANN or some similar body (sorry, I lost track of who “runs” the Internet) simply delete the IP addresses of the servers involved so the computers in their network cannot connect to the Internet?
Obviously they can use proxies but don’t the latter refer to some sort of master IP list to communicate with the “home servers”?
(Please bear with me, I’m not an IP professional)
Gary, simply put: no.
No there is no “master IP list” and no there is no central authority who can deny you access.
Incidentally I am not sure there is any way the Postal Service can prevent you from walking up to a mailbox and dropping a stamped envelope into it addressed to anyone else in the world. The inability to identify the murderer who mailed anthrax-laden envelopes to Congress is a good analogy, if you can’t identify the offender how can you take action against them?
Another analogy might be the difficulty in preventing repeat drunk driving offenders. What you suggest is very similar to trying to prevent such a driver from entering the public highway system by blockading all the driveways they might use.
The other issue is, who decides what is abusive enough to deny access? On what grounds, their personal prejudice?
Research the “Great Firewall of China” for some idea of the issues. Look into the attempts and problems the Iranians had in blocking tweets and emails and IMs during the post-election protests last year too.
Then think about the fact that maybe some of the network activity we consider malicious might in fact be acceptable if not embraced in other parts of the world. Remember that much of the activity is aimed at transferring value, theft in simple terms, and that while such activity is undesirable to the victims it is quite acceptable to the beneficiaries.
Could internet thievery be better viewed as economic (or class) warfare? Are some of the perpetrators being harbored by foreign countries similarly to the way pirates were licensed as privateers during the 18th century imperialist conflicts?
sorry for the rambling, just wanted to provide food for thought…
What I was referring to with the USPS is that under certain conditions you will not receive mail.
As regards punishing spammers, ICANN says, “To reach another person on the Internet you have to type an address into your computer – a name or a number. That address has to be unique so computers know where to find each other. ICANN coordinates these unique identifiers across the world. Without that coordination we wouldn’t have one global Internet.”
It seems to me that the body that makes it possible for others to connect with you should somehow be able to reverse the process. Who authorizes it to do so (the International Criminal Court?) is another matter
On a historical note, Brian, your graphic is way way nicer than anything produced at the Kefauver or Valachi Hearings.
I was just a baby at the time, but my older siblings (foodies all) still remark about the truly bad quality of the early “TV Dinners”. I can confirm that they didn’t get much better by the time of the Valachi Hearings. On St. Pat’s Day we eat Corned Beef & Cabbage; closing our eyes and thinking of “TV Dinners”
Does anyone know if Organized Cyber Crime officially “exists” as a finding of fact ? That was the point of these hearings, and I think the distinction between “Crime” and “Terror” is, if anything, more important to make today than in the past.
Kefauver Hearings
“1954 Hearings to Investigate Organized Crime in Interstate Commerce.”
McClellan (Valachi) Hearings
“Oct. 1963 Senate Permanent Subcommittee on Investigations”
Below quote is from the RSA FraudAction Research Lab blog:
” According to these networks’ “WHO IS” records, (all of which contain fake details), they are located in Kazakhstan, The Ukraine, Russia and Moldova”
The physical location, particularly the Kazakhstan location, has been disputed as also ‘fake detail’ by some security research.
How reliable is IP address geolocation data? Is all geolocation data still largely self-reported?
Even if the geo-location data is correct, it is probably the last hop to whatever bot(PC) the criminal has pwned.
Since the US and Korea have such high bandwidth, bots in America and South Korea are more valuable. The command and control are usually accounts that are paid with someone else’s ill gotten money, so they aren’t the location of the bot-herder either. Most criminals use VPN and/or SSL to communicate command and control to the crime “enterprise”. That way nobody really knows where they really are.
Law enforcement have been able to catch some of them buy shutting down enough of their web space and tricking them into taking a chance on direct control to get their web back up and running. A big mistake not may web criminals will make.
@t.w.
Quote= “…. The physical location, particularly the Kazakhstan location, has been disputed as also ‘fake detail’ by some security research. ..”
“How reliable is IP address geolocation data? Is all geolocation data still largely self-reported?”
End Quote
Much of it comes from “Regional Internet Registry” records combined with other sources, including guessing. It is reliable at the country level, and usually within regions. However if the operator is rogue and intent on masquerading, then it can be defeated. While the Troyak IP range may be assigned to Kazakhstan, the operator and C&C is based in Kiev, Ukraine. Remember that many of these “providers” are the actual cyber criminals themselves, or at best, are wholly ran as (CCAS) “Cyber Crime As a Service”
With respect to the cyber crime cesspool Troyak, and as I stated last week in this DSLR thread: http://www.dslreports.com/forum/r23933249-Zeus-botnets-suffer-mighty-blow-after-ISP-taken-offline When “Roman Starchenko” registered the matching troyak.org domain on 12/08/2009, he registered it to an address and phone number of an apartment in Kiev, Ukraine.:
Domain: TROYAK.ORG
Sponsoring Registrar:Directi Internet Solutions Pvt. Ltd.
d/b/a PublicDomainRegistry.com (R27-LROR)
Registrant ID:DI_10792377
Registrant Name:Roman Starchenko
Registrant Organization:Troyak
Registrant Street1:str. Miloslavskaya 17a, 75 ap.
Registrant Street2:
Registrant Street3:
Registrant City:Kiev
Registrant State/Province:Kiev
Registrant Postal Code:01001
Registrant Country:UA
Registrant Phone:+380.630231165
Registrant Phone Ext.:
Registrant FAX:
Registrant FAX Ext.:
Registrant Email:Staruy.rom@inbox.ru
Name Server: VETAXA.MANAGEDNS1.ESTBOXES.COM
Within a few hours of its birth the domain registration was changed to match the Troyak IP registry in Kazakhstan. Clearly an afterthought.
Also, take note that the one of the “Red Flagged” trojan criminal networks in Brian’s diagram: AS 50369 VISHCLUB is also registered to the same address in Kazakstan, and coincidentally the listed contact is using an @troyak.org email address:
Incidentally, up until mid February troyak.org was hosted on IP 195.93.184.1 which also hosted the nameserver ns.troyak.org and it served vishclub.net as well
That IP was also based in Kiev, Ukraine, also:
inetnum: 195.93.184.0 – 195.93.185.255
netname: ALYANSHIMIYA-NET
descr: Alyanshimiya LLC
country: UA
org: ORG-AL84-RIPE
admin-c: DS7111-RIPE
tech-c: IO303-RIPE
status: ASSIGNED PI
mnt-by: ALYANSHIMIYA-MNT
mnt-by: RIPE-NCC-HM-PI-MNT
mnt-lower: RIPE-NCC-HM-PI-MNT
mnt-routes: ALYANSHIMIYA-MNT
mnt-domains: ALYANSHIMIYA-MNT
source: RIPE # Filtered
organisation: ORG-AL84-RIPE
org-name: Alyanshimiya LLC
org-type: OTHER
descr: Alyanshimiya LLC
address: 15 Pecherskiy downstreet
address: Kiev, Ukraine
phone: +380 44 2511308
e-mail: info@udobreniya.com
admin-c: DS7111-RIPE
tech-c: IO303-RIPE
mnt-ref: ALYANSHIMIYA-MNT
mnt-by: ALYANSHIMIYA-MNT
source: RIPE # Filtered
IMO, from a security perspective, it is irresponsible not to block data at border routers, to or from these networks, or their peers, many do.
MGD
Thank you MGD – very interesting and enlightening!
MGD, I notice Hurrican Electric is listed at one of the articles you linked, as attempting to take remedial action in this war. Are they another innocent bystander in this mess? If they are; it would make sense, as they are listed as an ISP on one of the disreputable sites that evidence shows was involved in my case of internet ID theft.
The crooks may be using reputable ISP names in their web pages as a slap in the face of the very folks trying to bring them down. That actually makes sense in this way of thinking, as a motivation for the defacement and obfuscation of their where-abouts; and putting in a plug against their enemies.
looks like someone is paying attention to these guys in a more public way now…
http://twitter.com/asn_tracking
Thanks John Doe; everything helps in this information war.
Brian,
Very interesting interview. You should write a magazine article for The New Yorker, Vanity Fair or The Daily Beast about this. Then later a book.
Then you could go on 60 Minutes and The Daily Show. Then people would start to notice.
Good job.
He use to work for the Washington Post; they have lost a very good asset. I think he likes it better this way. Just spread the word about this valuable site!
Dear JCitizen,
I know where he worked and I followed his work there for years.
However, the WP has been having problems and a lot of their best people have bailed — including Brian.
Let’s let Brian figure out what he should do.
I just made a suggestion. I’ll support whatever he decides.
My apologies to Brian, I really wish him the best and I think he may get more exposure here. Newspapers are a dying medium.
My previous post was a stupid statement by me; I’m just too much of an incurable optimist I guess!
Sincerely
JCitizen
Re Manoa Kahuna’s publishing recommendations to Brian: I second Vanity Fair. It’s a personal prejudice, but I have been a subscriber since 1986. And that is because subjects of such interest and significance and writers of his caliber are a staple therein.
This network is currently routed by:
MnogoByte LLC
Roman Gordeev
Tushinskaya str., 12
125362 MOSCOW
RUSSIAN FEDERATION
phone: +74957392551
fax-no: +74956269784
e-mail: info@mnogobyte.ru
noc@mnogobyte.ru; abuse@mnogobyte.ru
Someone should sent them some email telling them they are facilitating cyber crime by allowing service to these networks
@infosec_pro: If you piss off NANOG, that’s a quick way to get blacklisted from the internet.
What I do most often is setup ACLs for my DNS servers, so that IPs on the following pages get denied automatically or spoofed to a honeypot server:
http://mtc.sri.com/
http://www.emergingthreats.net/index.php/rules-mainmenu-38.html
Brian,
As always, thanks for your incredible threat analysis and reporting. An earlier fan mentioned vanity fair, which made me think about another avenue to share your insights; Reader’s Digest. They have articles exposing scams from time to time and have regular articles by health professionals, etc. I think timely articles from you in that media would also be well received and reach a greater audience.
Keep up the good work.
@Tyler: That’s actually a good idea. The stuff posted here is great, but Brian is preaching to the choir. Readers Digest reaches an entirely different demographic.
They’ve got a slide show on their website right now about keeping yourself safe online, but it’s very superficial generalities. It won’t change anyone’s behavior, because people already know those things and think they only apply to other people. Brian can tell stories about real people who have been harmed. That’s more engaging reading, and it teaches more effectively.
You can shut down all the C&C servers you want, but if people keep clicking on ecards and fake password updates, there will always more botnets to take their places. If everyone who subscribes to RD knew how to look up the registration whois for a domain name to find out whether “paypal-registration.com” belongs to Paypal — or whether it only came into being two days ago using a privacy protected registration — it would drastically cut down the number of people falling victim to phishing.