Hawk.js is an advanced, open-source sitemap generator and SEO automation tool that simplifies the SEO process for web developers, SEO professionals, and digital marketers. It automates sitemap creation and submission to major search engines like Google, Bing, Yahoo, Yandex, and others supporting IndexNow.

Hawk.js offers several key benefits:

• Automated Sitemap Creation: Easily generate and maintain sitemaps with minimal manual effort.
• Optimized Search Engine Indexing: Ensure your website is indexed quickly and accurately, improving its visibility and ranking.
• Streamlined Submissions: Automates the process of submitting your sitemaps to major search engines, resulting in faster indexing.

Some of the key features of Hawk.js include:

• Automated Sitemap Generation: Automatically generate sitemaps with ease, saving time and reducing manual work.
• SEO Optimization: Hawk.js helps ensure that your site is correctly indexed by search engines, boosting visibility and improving rankings.
• Real-Time Sitemap Updates: Keep your sitemaps current, reflecting the latest changes to your website content.
• Highly Customizable Configurations: Tailor Hawk.js to meet your specific needs with a variety of configuration options.
• Multi-Search Engine Support: Submit your sitemaps to multiple search engines such as Google, Bing, Yahoo, Yandex, DuckDuckGo, and others supporting IndexNow for global reach.

Hawk.js helps you:

• Save Time and Effort: Automate repetitive tasks like generating and submitting sitemaps, so you can focus on more important tasks.
• Enhance Your SEO Performance: Ensure faster and more accurate indexing, improving your site’s visibility and ranking in search results.
• Full Customization: Fine-tune your sitemap and submission strategies based on your needs.
• Stay Up-to-Date with SEO Standards: Hawk.js supports IndexNow, ensuring your website is competitive and adheres to the latest SEO trends.

The latest version of Hawk.js (1.5.0) includes several updates:

• Official Hawk.js Logo: A new logo has been integrated for better brand recognition.
• TypeScript Support: The init command now supports TypeScript for easier configuration management.
• Updated Configuration Format: The config format has moved from JSON to JavaScript for better flexibility.
• Expanded Search Engine Strategies: Additional strategies like Google Webmaster Tools and IndexNow have been added.
• Enhanced Performance: Performance optimizations have been made for improved speed and usability.

Yes, Hawk.js is an open-source project released under the Apache 2.0 License. You can freely use and modify it for both personal and commercial purposes. However, make sure to provide proper attribution as per the license terms.

Hawk.js was developed by Darsan, founder of Cresteem, a web development company. It is an open-source project designed to help web developers and SEO professionals streamline SEO tasks.

To use Hawk.js, you need the following prerequisites:

• Node.js (v18.0 or later) installed on your system. You can download it from Node.js website.
• Internet connection to install Hawk.js from the npm registry.

You can install Hawk.js using your preferred package manager. Here’s how:

• Using Yarn:

yarn add @cresteem/hawk-js -D

• Using npm:

npm install @cresteem/hawk-js -D

To create a default configuration file, run the following command:

hawk init

This will generate a hawk.config.js file in your project directory.

After generating the hawk.config.js file, open it and define the patterns you wish to include or exclude. You can also customize additional settings as per your project’s requirements. Learn more in the Configuration Guide.

For certain indexing strategies, you will need to provide specific credentials:

• Google Webmaster Tools credentials for G-Webmaster, G-Webmaster2, and G-Index strategies.
• FTP credentials for the IndexNow strategy. Be sure to input these credentials in the configuration file based on the strategy you choose. Learn more about credentials in the Google Credentials and FTP Credentials sections.

To run Hawk.js with a specific strategy, use the command:

hawk --strategy <number>

Where <number> represents the following strategies:

• 1 for Index-Now
• 2 for G-Webmaster
• 3 for G-Webmaster2
• 4 for G-Index

You can also use optional parameters like --include, --exclude, and --prettify. For more information, refer to the CLI References.

After setting up Hawk.js, the next step is to customize it for your project. Visit the Configuration Details page to:

• Learn how to edit the hawk.config.js file.
• Define URLs to include or exclude from the sitemap.

This will ensure Hawk.js is tailored to your SEO needs and runs smoothly for your website.

The default configuration for Hawk.js is defined in the hawk.config.js file. Here’s the template:

/** @type {import("@cresteem/hawk-js").hawkJsOptions} */
const config = {
  lookupPatterns: ["**/*.html", "**/*.htm"],
  ignorePattern: ["node_modules/**"],
  timeZone: "Asia/Kolkata",
  domainName: "www.cresteem.com",
  sitemapPath: "sitemap.xml",
  robotPath: "robot.txt",
  serviceAccountFile: "gserv.json",
  ftpCredential: { username: "", password: "", hostname: "" },
};
 
exports.default = config;

You can customize this template to suit your website’s SEO needs.

Here’s a breakdown of the default configuration options:

To configure Hawk.js, follow these steps:

1. Create hawk.config.js:
   • Run the CLI command:

     hawk init
   • This will generate the hawk.config.js file in your project’s root directory.
2. Customize the Settings:
   • Modify the configuration values to meet your project’s requirements (e.g., file patterns, time zone, credentials).
3. Save the Changes:
   • After editing the hawk.config.js file, save it to apply the new configuration.

For specific indexing strategies, the following credentials are required:

• FTP Credentials: Required for strategies like IndexNow, G-Webmaster, and G-WebMaster2. You will need to provide username, password, and hostname for FTP access to upload the sitemap.
• Service Account File: Required for G-Webmaster, G-WebMaster2, and G-Index strategies. This JSON file contains your Google service credentials to integrate with Google Search Console.

Learn more in the Google Credentials and FTP Credentials sections.

Once you’ve configured Hawk.js, the next step is to explore supported indexing strategies. Learn about these strategies in the Effective Indexing with Hawk.js section to optimize your website’s SEO and indexing process.

IndexNow is a protocol designed to notify search engines instantly when new or updated content is available on your website, significantly speeding up the indexing process. Hawk.js integrates with IndexNow to automate the process of notifying search engines like Bing, Yahoo, Yandex, and DuckDuckGo about content changes on your website.

Here’s how it works:

1. Triggering Hawk.js with IndexNow: Hawk.js checks for updates since the last execution and identifies any changed files.
2. Tracking Updates: Only files that have been updated are marked for indexing.
3. Authentication: Hawk.js generates a unique hash key and stores it on an FTP server, which is used for authenticating the request with the IndexNow API.
4. API Request: Hawk.js sends a request to the IndexNow API with a list of updated files.
5. Search Engine Indexing: Supported search engines crawl the updated files, ensuring that the content is quickly indexed and made available in search results.

For more details, refer to the IndexNow website.

IndexNow is most beneficial when:

• You need rapid indexing of newly updated or added content, such as product pages, blog posts, or news articles.
• Your website’s content changes frequently, and you want real-time updates in search engines.
• You want to ensure your website is indexed instantly without relying on traditional crawling methods, which can take time.

Make sure to verify that the search engines you are targeting support IndexNow, as it is supported by Bing, Yahoo, Yandex, Sheznam.cz, Naver, Yep, and DuckDuckGo.

To use IndexNow with Hawk.js, follow these steps:

Prerequisites:

• FTP Server: Set up an FTP server to store the hash key generated by Hawk.js for authentication.
• Configuration: Ensure that your hawk.config.js includes the correct FTP credentials and paths.

Steps:

1. Configure Your FTP Server: Set up your FTP server to store the hash key file, which is used for authentication with the IndexNow API.

2. Run Hawk.js with IndexNow: To trigger the IndexNow strategy, run the following command in your terminal:

   hawk --strategy 1

3. Monitor Updates: Hawk.js will track the modification timestamps of files and send only the updated or newly added files to the IndexNow API.

4. Verify Successful Indexing: Once the IndexNow API processes the request and crawls the updated pages, your content will be indexed by the search engines, ensuring it’s up-to-date in search results.

Ensure that your FTP credentials are correctly configured in the hawk.config.js file, and that your FTP server is ready to store the generated hash key.

Here are a few key considerations for using IndexNow with Hawk.js:

• FTP Credentials: Ensure your FTP credentials are configured correctly in the hawk.config.js file to enable communication with your FTP server.
• Authorization: The first time you trigger Hawk.js with IndexNow, a unique hash key will be generated and stored on your FTP server for authentication with the IndexNow API.
• Search Engine Support: Double-check that the search engines you want to target support the IndexNow protocol, as not all search engines may be compatible.

Both GWebMaster and GWebMaster2 are strategies in Hawk.js for submitting sitemaps to Google via the Google Webmaster API. However, there are some key differences:

• GWebMaster: Submits the generated sitemap to Google Search Console, enabling Google to crawl and index the content.
• GWebMaster2: Includes all the functionality of GWebMaster but with additional features, such as fetching feedback from Google Search Console, including indexing status (number of indexed pages), crawl errors, and the last crawl timestamp.

The key difference is that GWebMaster2 provides detailed feedback on the indexing status and any issues Google encountered during the crawl, while GWebMaster only handles sitemap submission.

To use GWebMaster or GWebMaster2 strategies with Hawk.js, you need the following:

• FTP Credentials: These are necessary to upload the sitemap to your FTP server, where Google Search Console can access it.
• Google Service Account File (gserv.json): This file is required to authenticate Hawk.js with Google Search Console for both GWebMaster and GWebMaster2 strategies.

Ensure that both sets of credentials are configured properly in your hawk.config.js file for the integration to work correctly.

Follow these steps to set up GWebMaster or GWebMaster2 strategies with Hawk.js:

1. Generate Google Service Account File:

   • Visit the Google Credentials page for detailed instructions on obtaining a Google Service Account File (gserv.json).

2. Configure Hawk.js with Service Account and FTP Credentials:

   • Update your hawk.config.js file with the Google Service Account File and FTP credentials:

   serviceAccountFile: "gserv.json",
   ftpCredential: { 
     username: "your-ftp-username", 
     password: "your-ftp-password", 
     hostname: "your-ftp-hostname" 
   }

3. Run Hawk.js with GWebMaster or GWebMaster2 Strategy:

   • To trigger the GWebMaster strategy, run the following command:

   hawk --strategy 2
   • To trigger the GWebMaster2 strategy, run:

   hawk --strategy 3

4. Monitor Submission and Indexing Status:

   • After submitting the sitemap, you can check your Google Search Console for updates on indexing and crawl status.
   • With GWebMaster2, you can also receive detailed feedback, such as the number of indexed pages and any crawl errors.

Using the Google Webmaster API (through GWebMaster and GWebMaster2) with Hawk.js offers several benefits:

• Automated Sitemap Submission: Automatically generate and submit your sitemap to Google Search Console, eliminating the need for manual intervention.
• Faster Indexing: Ensure that your content is indexed more quickly by Google, improving SEO performance.
• Detailed Crawl Feedback: With GWebMaster2, you can receive valuable feedback about your site’s indexing status, crawl errors, and more.
• Error Monitoring: Use the feedback from GWebMaster2 to identify and resolve crawl issues, improving your site’s visibility in search results.

These features help streamline the process of managing your website’s SEO and indexing status with Google Search Console.

G-Index is a strategy in Hawk.js that utilizes the Google Indexing API to notify Google about new or updated content on your website. This strategy is particularly useful for ensuring that time-sensitive or frequently updated pages, such as job postings, livestream events, and breaking news, are indexed quickly by Google. By using G-Index, Hawk.js allows faster indexing and better search visibility for such content.

G-Index is most effective in the following scenarios:

• Time-Sensitive Content: For content that needs immediate visibility, like job postings or livestream videos.
• Short-Lived Content: For content with a brief relevance period, such as promotions or temporary events.
• Frequently Updated Content: For dynamic websites that require continuous updates, ensuring the latest content is indexed efficiently.

Here’s how G-Index works in Hawk.js:

1. File Tracking: Hawk.js checks the last run timestamps and compares them with file modification times. It then notifies Google of any updated files.

   • On the first run, all files matching the defined patterns are included.
   • On subsequent runs, only newly updated or added files are submitted.

2. Authentication: The Google Indexing API requires authentication via a service account JSON file, which must be set up and configured in Hawk.js.

3. API Submission: After collecting updated routes, Hawk.js submits them to the Google Indexing API, prompting Google to crawl and index those URLs faster.

To use G-Index with Hawk.js, the following are required:

• Service Account File: This file is necessary for authenticating the API requests. You can obtain it from the Google Cloud Console. Detailed steps can be found in our Google Credentials Setup Guide.

To use G-Index in Hawk.js, follow these steps:

1. Update Configuration: Modify your hawk.config.js file with the necessary file patterns and service account details:

/** @type {import("@cresteem/hawk-js").hawkJsOptions} */
const config = {
    lookupPatterns: ["**/*.html", "**/*.htm"],
    ignorePattern: ["node_modules/**"],
    domainName: "www.example.com",
    serviceAccountFile: "path/to/your-service-account.json",
};
exports.default = config;

2. Run Hawk.js with G-Index Strategy: Trigger the G-Index strategy by running the following command:

hawk --strategy 4

3. Monitor API Responses: After submission, check your CLI logs for details on the routes submitted and confirm indexing success via Google Search Console.

Here are some practical use cases for G-Index:

• Job Portals: Automatically notify Google when new job listings are posted, ensuring quick indexing of time-sensitive job data.
• Livestream Platforms: Use the G-Index strategy to ensure Google indexes event pages with BroadcastEvent data efficiently, so users can find them in search results as soon as the event starts.
• News Websites: Submit updates for breaking news articles, ensuring that the latest headlines are indexed immediately for visibility.

Before you begin obtaining Google service account credentials for Hawk.js, make sure you have:

• A Google account
• Access to the Google Cloud Console
• Admin access to your Google Search Console property

Here is a detailed, step-by-step guide to obtain Google service account credentials for Hawk.js:

1. Open the Google Cloud Console.
2. Create a new project by clicking the project dropdown at the top left and selecting “New Project”.
3. Enter a name for your project and click “Create”.

1. Go to APIs & Services > Dashboard in the left-hand navigation menu.
2. Enable the Indexing API by searching for it, clicking on it, and selecting Enable.
3. Similarly, enable the Search Console API by searching for it and selecting Enable.

1. Navigate to IAM & Admin > Service Accounts.
2. Click on + Create Service Account.
3. Add a name and description for the service account, then click Create.
4. Grant the service account the Owner role and click Continue, then Done.

1. Find the service account you created and click on its email.
2. Go to the Keys tab, then click Add Key > Create New Key.
3. Select the JSON option and click Create. This will download a JSON key file.
4. Store this file securely as gserv.json in your web project root directory.

1. Go to Google Search Console.
2. Select your property and go to Settings > Users and permissions > Add user.
3. Enter the email address of your service account and set the role to Owner, then click Add.

Once you have completed the steps to obtain your Google service account credentials, store the downloaded JSON key file (gserv.json) securely in your web project’s root directory. This file will be used by Hawk.js for authenticating API requests to both the Google Indexing API and Google Search Console API.

For detailed assistance, refer to the Google Cloud Console Documentation and Google Search Console Help.

When configuring FTP credentials for Hawk.js, keep these important considerations in mind:

• FTP Root Directory: The FTP root directory must align with the web server’s root directory (e.g., public_html or /var/www/html).
• No Directory Navigation: Hawk.js does not navigate between directories on your FTP server. It only uploads files directly to the home (primary) directory of the FTP connection.

Here are the steps to set up an FTP server and integrate it with Hawk.js:

• Ensure that your hosting plan includes FTP support by reviewing your plan’s features or contacting your hosting provider.

• Use your hosting provider’s control panel (e.g., cPanel, HPanel for Hostinger, or a custom dashboard) to create an FTP account.

• When setting up your FTP account, define these credentials:
  • Hostname: The domain name or IP address of the FTP server (e.g., ftp.example.com or 192.168.1.1).
  • Username: The username for your FTP account.
  • Password: The password for your FTP account.

To integrate FTP credentials into Hawk.js, update your hawk.config.js file with the necessary FTP details:

/** @type {import("@cresteem/hawk-js").hawkJsOptions} */
const config = {
    ftpCredential: {
        username: "your-ftp-username",
        password: "your-ftp-password",
        hostname: "ftp.example.com",
    },
};
exports.default = config;

This will ensure that Hawk.js can use the FTP credentials for uploading files such as sitemaps and authorization files.

Follow these best practices when configuring FTP for Hawk.js:

1. Match FTP and Web Server Roots: Ensure the FTP root directory matches the web server’s root directory (e.g., /public_html).
2. Secure Your Credentials: Store your FTP credentials securely and restrict access to the hawk.config.js file to prevent unauthorized access.
3. Verify File Uploads: After setting up Hawk.js and running a strategy that requires FTP (e.g., IndexNow), verify that the files are correctly uploaded to the FTP server’s root directory.

Here’s a table of common hosting providers and their FTP support:

To get started with the Hawk.js CLI, run the following command:

hawk [options] [command]

Ensure Hawk.js is installed either globally or locally in your project. If you’re not sure, install it globally using:

npm install -g @cresteem/hawk-js

Here are the available global options in the Hawk.js CLI:

The Hawk.js CLI supports the following strategies:

Use the genmap command to generate a sitemap based on your specified patterns. You can also upload it to an FTP server using the --commit option:

Example for generating a sitemap:

hawk genmap -i "src/**/*.html" -e "node_modules/**"

Example for generating and uploading a sitemap:

hawk genmap -i "src/**/*.html" -c

To initialize a default configuration file (hawk.config.js), use the init command:

hawk init

After running this command, the CLI will output:

🚀 Hawk.js configuration initialized.

No, the Hawk.js CLI supports only one strategy per command. To use multiple strategies, you’ll need to run separate commands for each strategy.

If no valid strategy is provided, the CLI will prompt you to choose one of the supported strategies:

⭕ Must choose any one number from below:
1. IndexNow
2. G-WebMaster
3. G-WebMaster2
4. G-Index

To get started with the Hawk.js API, import the Hawk class and create an instance of it:

import { Hawk } from "hawkjs";
const hawk = new Hawk();

Hawk.js provides several useful methods for sitemap generation and indexing automation:

1. hawk(strategy, lookupPatterns, ignorePatterns, prettify): Executes sitemap generation and submission based on a specified strategy.

2. utils.makeSitemap(lookupPatterns, ignorePatterns, prettify, upload): Generates a sitemap and uploads it to the FTP server.

3. utils.makeRobot(): Creates a robots.txt file to guide search engine crawlers.

4. utils.getLastRunTimeStamp(): Retrieves the timestamp of the last successful run.

5. utils.getUpdatedRoutesPath(lastRunTimeStamp, lookupPatterns, ignorePatterns): Finds updated file paths based on the last run timestamp.

The hawk method is the main function for generating sitemaps and submitting them based on the selected strategy. Example usage:

await hawk.hawk("IndexNow", ["src/**/*.html"], ["node_modules/**"], true);

• strategy: Choose from IndexNow, GWebmaster, GWebmaster2, or GIndex.
• lookupPatterns: Array of file patterns to include in the sitemap.
• ignorePatterns: Array of file patterns to exclude from the sitemap.
• prettify: Boolean to prettify the generated sitemap.xml.

The makeSitemap method generates a sitemap.xml file based on the provided patterns. Example usage:

const sitemapStatus = await hawk.utils.makeSitemap(
  ["src/**/*.html"],
  ["node_modules/**"],
  true,
  true
);
console.log(sitemapStatus);

• lookupPatterns: Array of file patterns to include.
• ignorePatterns: Array of file patterns to exclude.
• prettify: Boolean to prettify the sitemap.xml.
• upload: Boolean to upload the sitemap to an FTP server.

You can create a robots.txt file with the following method:

const robotTxtStatus = hawk.utils.makeRobot();
console.log(robotTxtStatus);

Use the getLastRunTimeStamp method to retrieve the timestamp:

const lastRunTimeStamp = hawk.utils.getLastRunTimeStamp();
console.log("Last Run Timestamp:", lastRunTimeStamp);

You can use getUpdatedRoutesPath to find updated routes since the last run:

const updatedRoutes = hawk.utils.getUpdatedRoutesPath(
  lastRunTimeStamp,
  ["src/**/*.html"],
  ["node_modules/**"]
);
console.log("Updated Routes:", updatedRoutes);

Hawk.js loads configuration options from the hawk.config.js file. You can specify various settings like lookupPatterns, ignorePattern, and FTP credentials.

Example configuration:

module.exports = {
  lookupPatterns: ["src/**/*.html"],
  ignorePattern: ["node_modules/**"],
  ftpCredentials: {
    hostname: "ftp.example.com",
    username: "yourUsername",
    password: "yourPassword",
  },
  googleServiceAccount: "./service-account.json",
};

Access the configuration via the configurations property:

console.log(hawk.configurations.lookupPatterns);

For more details on each supported strategy, refer to the strategy-specific documentation:

• IndexNow Strategy
• Google Webmaster Strategies
• Google Indexing API (GIndex)

Hawk.js can be integrated into the following CI/CD platforms:

• GitHub Actions
• GitLab CI
• Jenkins

Each platform has specific workflow templates and configurations to support Hawk.js functionalities.

In CI/CD pipelines, Hawk.js performs the following tasks:

1. Build Your Website: Compiles and prepares your website for deployment.
2. Generate Sitemaps: Creates sitemap.xml and robots.txt files using Hawk.js.
3. Trigger Search Engines: Notifies search engines via different strategies:

• IndexNow: For Bing, Yahoo, Yandex, and other IndexNow-compatible engines.
• Google Webmaster API: For submitting and managing sitemaps through Google Search Console.
• Google Indexing API: Notifies Google about job postings or live stream updates.

4. Optional: You can choose to generate sitemaps only without notifying search engines.

Hawk.js supports the following strategies for notifying search engines:

• IndexNow: Used for notifying IndexNow-compatible search engines like Bing and Yandex.
• Google Webmaster: Submits and manages sitemaps via Google Search Console.
• Google Indexing API: Used for notifying Google about page updates, including live media or job postings.

Hawk.js offers customizable workflows for the following actions:

1. IndexNow Strategy: Notifies IndexNow-compatible search engines.
2. Google Webmaster Strategy: Submits sitemaps to Google Search Console.
3. Google Indexing API Strategy: Notifies Google about job postings or live stream updates.
4. Sitemap Generation Only: Generates sitemaps without submitting them to search engines.

Each platform (GitHub Actions, GitLab CI, Jenkins) provides tailored workflow templates and configurations, which can be customized based on your needs.

You can find platform-specific workflow examples and templates in the following pages:

• GitHub Actions Integration
• GitLab CI Integration
• Jenkins Integration

These pages provide step-by-step guides to integrate Hawk.js into your chosen CI/CD platform.