A while ago, Simone (@evilsocket) published his awesome ditto tool for checking a domain for active imposter attacks, and I finally found some time to play with it.
In this blog post, I’ll share some insights from initial experiments and recon, and open source the toolchain I created to run a central ditto Transform server to analyze the data in Maltego.
With the ditto Transform server, multiple analysts can run Transforms to check domains for active imposters without calculating the variations and checking them on their own machine, but on a dedicated server instead. The server can take care of throttling the external lookups to avoid being rate limited. In addition, future updates could add a queuing mechanism.
Releasing DittoTRX Transforms for Maltego 🔗︎
This 2-part article series will walk you through the release of the ditto Transform server, setting up the server, and how to use the Transforms in Maltego.
IDN Homograph Attacks 🔗︎
Internationalized domain name (IDN) homograph attacks exploit the fact that characters from different alphabets often look alike the ones from the Latin alphabet (= they are homographs). A malicious actor can use this to deceive a victim, for example, to visit a malicious website that pretends to be the legitimate one, a classic phishing attack. A regular user of Facebook may be tricked into to clicking a link where the Latin character “a” is replaced with the Cyrillic character “а”.
You can read more about IDN homograph attacks on Wikipedia.
Maltego Code for A Custom Transform Server 🔗︎
I’ll be demoing the use of my new Maltego golang package here and show how to create a simple Transform server for any tool in very little time.
At the core, all we need is an HTTP service that replies to the request from the Maltego Transform service.
The Maltego package provides a primitive to register http handlers:
func RegisterTransform(handlerFunc http.HandlerFunc, name string)
Since the ditto tool invocations can take longer depending on configuration and your hardware, I configured the http service to never time out. Querying all registered and live imposter domains for common names like amazon etc, without varying TLDs, took between 2-3 minutes for me.
To handle requests to the root page, I used the
func Home(w http.ResponseWriter, r *http.Request)
handler, which will give a simple text based overview of the registered handlers and their routes once visited.
func main() {
flag.Parse()
const (
// ditto status names
registered = "registered"
available = "available"
)
// register transforms to http.DefaultServeMux
// all similar domains
maltego.RegisterTransform(ditto("", false), "similarDomains")
// only registered domains
maltego.RegisterTransform(ditto(registered, false), "registeredDomains")
// only registered domains that resolve to an IP
maltego.RegisterTransform(ditto(registered, true), "liveDomains")
// only show domains that are available and not registered
maltego.RegisterTransform(ditto(available, false), "availableDomains")
// only live domains that resolve for all TLDs
maltego.RegisterTransform(ditto(registered, true, "-tld"), "liveDomainsTLD")
// register catch all handler to serve home page
http.HandleFunc("/", maltego.Home)
fmt.Println("serving at", *flagAddr)
s := &http.Server{
Addr: *flagAddr,
Handler: http.DefaultServeMux,
ReadTimeout: 0,
ReadHeaderTimeout: 0,
WriteTimeout: 0,
IdleTimeout: 0,
MaxHeaderBytes: 0,
}
// start server
err := s.ListenAndServe()
if err != nil {
log.Fatal("failed to serve HTTP: ", err)
}
}
Let’s look at the implementation of ditto invocations:
var ditto = func(status string, hasIP bool, args ...string) http.HandlerFunc {
return maltego.MakeHandler(func(w http.ResponseWriter, r *http.Request, t *maltego.Transform) {
// get host that was queried
host := t.RequestMessage.Entities.Items[0].Value
fmt.Println("got request from", r.RemoteAddr, "to lookup:", host)
if !govalidator.IsDNSName(host) {
t.AddUIMessage("invalid domain: "+host, maltego.UIMessageFatal)
return
}
id, err := cryptoutils.RandomString(10)
if err != nil {
fmt.Println("failed to generate id:", err)
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
file := filepath.Join(storage, id+"-"+host)
finalArgs := []string{"-domain", host, "-no-progress-bar", "-csv", file, "-throttle=1000", "-workers=4", "-whois"}
for _, a := range args {
finalArgs = append(finalArgs, a)
}
start := time.Now()
// run ditto
// we are running inside a docker container, the ditto binary has been copied into it at build time.
// TODO: drop privileges
out, err := exec.Command("/root/ditto", finalArgs...).CombinedOutput()
if err != nil {
fmt.Println(string(out))
fmt.Println("failed to run ditto:", err)
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
data, err := ioutil.ReadFile(file)
if err != nil {
fmt.Println("failed to read file:", err)
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
rd := csv.NewReader(bytes.NewReader(data))
records, err := rd.ReadAll()
if err != nil {
fmt.Println("failed to read CSV records:", err)
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
fmt.Println("results for", host, "=", len(records), "in", time.Since(start))
// process results
for i, rec := range records {
//fmt.Println(rec)
if i == 0 {
// skip CSV header
continue
}
// handle status if provided
if status != "" {
if rec[2] != status {
continue
}
}
if hasIP {
if rec[3] == "" {
continue
}
}
addEntity(t, rec)
}
_ = os.Remove(file)
})
}
As you can see, the ditto function is a wrapper to invoke the ditto commandline tool with different flag combinations, by wrapping the
func MakeHandler(handler func(w http.ResponseWriter, r *http.Request, t *Transform)) http.HandlerFunc
primitive to construct transform handlers.
Ditto generates a CSV file as output, which is read, processed, and afterwards deleted again. Some simple filtering is applied based on the desired status or the existence of an IP address before collecting the entities.
Adding entities is done using a simple helper function that takes the CSV output from ditto and assigns the fields to the correct properties:
func addEntity(t *maltego.Transform, rec []string) {
e := t.AddEntity("dittotrx.IDNDomain", rec[0])
e.AddProp("unicode", rec[0])
e.AddProp("ascii", rec[1])
e.AddProp("status", rec[2])
e.AddProp("ips", rec[3])
e.AddProp("names", rec[4])
e.AddProp("registrar", rec[5])
e.AddProp("created_at", rec[6])
e.AddProp("updated_at", rec[7])
e.AddProp("expires_at", rec[8])
e.AddProp("nameservers", rec[9])
}
Now that we have some logic for the server side, we need to create a client side Maltego configuration.
Running Maltego Transforms will produce output Entities on a Maltego graph, which are modeled based on the data that ditto provides. In order to further drill down on their data, we will also add a couple of local Transforms. These can be found in cmd/transform:
$ tree cmd/transform/
├── lookupAddr
│ └── main.go
├── toDateCreatedAt
│ └── main.go
├── toDomains
│ └── main.go
├── toIPAddresses
│ └── main.go
├── toNameServers
│ └── main.go
├── toRegistrars
│ └── main.go
└── visitDomain
└── main.go
7 directories, 7 files
Let’s look at the implementation of a simple local Transform, as a standalone binary:
// This is an example for a local Transform that does a reverse name lookup for a given address.
// It will take an IP address and return the hostnames associated with it as Maltego Entities.
func main() {
log.Println(os.Args[1:])
// parse arguments
lt := maltego.ParseLocalArguments(os.Args[1:])
// ensure the provided address is valid
ip := net.ParseIP(lt.Value)
if ip == nil {
maltego.Die("invalid ip", lt.Value+" is not a valid IP address")
}
// lookup provided ip address
names, err := net.LookupAddr(lt.Value)
if err != nil {
maltego.Die(err.Error(), "failed to lookup address")
}
// create new Transform
t := maltego.Transform{}
// iterate over lookup results
for _, host := range names {
e := t.AddEntity(maltego.DNSName, host)
e.AddProperty("hostname", "Hostname", maltego.Strict, host)
}
t.AddUIMessage("complete", maltego.UIMessageInform)
// return output to stdout and exit cleanly (exit code 0)
fmt.Println(t.ReturnOutput())
}
Parsing the commandline args provided by Maltego is done with maltego.ParseLocalArguments. I discard the first element in the args array via slicing because it is the name of the program.
We can then access the main value the Transform shall operate on from lt.Value and try to parse it as an IP. If that fails, the Transform can’t proceed and execution is stopped via maltego.Die, which will exit the program and display an error message to the user in Maltego. If everything is fine, we perform the lookup, construct a new maltego.Transform, add the resulting names as maltego.DNSName Entities (there are constants for the core Maltego types provided in my Maltego go package), and exit the program cleanly using t.ReturnOutput(). That’s it—Less than 20 lines of code and you have a local Transform!
Let’s compile the local Transforms and move their binaries to /usr/local/bin:
go build -o /usr/local/bin/lookupAddr cmd/transform/lookupAddr/*.go
go build -o /usr/local/bin/toDateCreatedAt cmd/transform/toDateCreatedAt/*.go
go build -o /usr/local/bin/toDomains cmd/transform/toDomains/*.go
go build -o /usr/local/bin/toIPAddresses cmd/transform/toIPAddresses/*.go
go build -o /usr/local/bin/toNameServers cmd/transform/toNameServers/*.go
go build -o /usr/local/bin/toRegistrars cmd/transform/toRegistrars/*.go
go build -o /usr/local/bin/visitDomain cmd/transform/visitDomain/*.go
I’ve created a small helper tool to generate Maltego Entities and Transforms based on a YAML configuration since configuring everything via the UI can be tedious. I’ll cover the details of this tool in another blog post. For now, we will just look at a simple example.
The following yaml is used to generate the Maltego configuration:
org: DittoTRX
author: Philipp Mieden
description: Transformations for the ditto tool
entities:
- name: IDNDomain
image:
name: domain
color: black
description: A domain likely used for phishing
parent: maltego.Domain
fields:
- name: unicode
description: Unicode representation of domain name
- name: ascii
description: ASCII representation of domain name
- name: status
description: Registration status of domain name
- name: ips
description: IPs for the domain name
- name: names
description: DNS names that resolve to this domains IPs
- name: registrar
description: The name of the registrar
- name: created_at
description: The date of creation
- name: updated_at
description: The last update
- name: expires_at
description: The date of expiry
- name: nameservers
description: Nameservers associated with domain
workingDir: ~
transforms:
- id: LookupAddr
input: maltego.IPv4Address
description: Lookup Address
executable: /usr/local/bin/lookupAddr
- id: ToDomainNames
input: dittotrx.IDNDomain
description: To Domain Names
executable: /usr/local/bin/toDomains
- id: ToRegistrarNames
input: dittotrx.IDNDomain
description: To Registrar Names
executable: /usr/local/bin/toRegistrars
- id: ToNameServers
input: dittotrx.IDNDomain
description: To Name Servers
executable: /usr/local/bin/toNameServers
- id: ToCreationDate
input: dittotrx.IDNDomain
description: To Creation Date
executable: /usr/local/bin/toDateCreatedAt
- id: ToIPAddresses
input: dittotrx.IDNDomain
description: To IP Addresses
executable: /usr/local/bin/toIPAddresses
- id: VisitDomain
input: dittotrx.IDNDomain
description: Open Domain in default Browser
executable: /usr/local/bin/visitDomain
As you can see, the Transforms all operate on our custom entity type IDNDomain and have a description and an executable set.
To generate the config, all we need to run is:
$ maltego-gen maltego.yml
material icon repository exists, pulling
bootstrapped configuration archive for Maltego
packing maltego dittotrx archive
packed maltego dittotrx archive
copied generated file to /Users/you/DittoTRX.mtz
The resulting .mtz archive can now be imported with Maltego:
Now all that is left is to head to your Transform Distribution Server and add the Transforms. For simplicity, I am using a basic authentication, which is protected over TLS. Future versions could use oauth as suggested by Maltego.
You can read more about how to set up a TDS or add Transforms in the excellent Maltego docs or Maltego’s Complete Guide for Building Custom Maltego Integrations.
Using Ditto Transforms in Maltego 🔗︎
In Part 2 of this article series, I will demonstrate how you can use these ditto Transforms in Matlego for domain investigations.
Follow Maltego on Twitter and LinkedIn and subscribe to Maltego’s email newsletter to stay updated on the latest releases, tutorials, and webinars!
About the Author 🔗︎
Philipp Mieden
Philipp Mieden is a security researcher and software engineer from Germany, currently focusing on network security monitoring and the use of machine learning. He presented his research on classifying malicious behavior in network traffic at several international contests from Kaspersky Lab and won multiple prizes. After finishing his bachelor at the LMU Munich, he moved to Amsterdam to continue his master studies in “Security and Network Engineering” at the UvA. He is sharing many of his projects on Github, but you can also find him on ResearchGate or Twitter. Besides network monitoring, Philipp is also interested in hardware security, industrial control systems and reverse engineering.
