I have purchased a number of deep fryers over the years. This one is the best by far. It can fry 1.98 pounds of food at a time. It works extremely well. The only features that it does not have and I don't think either is necessary, is a timer and a drain spout for removing the oil.
jalsan 19191a764c -black-reloaded-serial-number-download[ -black-reloaded-serial-number-download ][ -black-reloaded-serial-number-download ][ -black-reloaded-serial-number-download ]link= -black-reloaded-serial-number-downloadlink= -black-reloaded-serial-number-downloadlink= -black-reloaded-serial-number-download
deep black reloaded serial number generator
On compaction: unlike the older message formats, magic v2 and above preserves the first and last offset/sequence numbers from the original batch when the log is cleaned. This is required in order to be able to restore the producer's state when the log is reloaded. If we did not retain the last sequence number, for example, then after a partition leader failure, the producer might see an OutOfSequence error. The base sequence number must be preserved for duplicate checking (the broker checks incoming Produce requests for duplicates by verifying that the first and last sequence numbers of the incoming batch match the last from that producer). As a result, it is possible to have empty batches in the log when all the records in the batch are cleaned but batch is still retained in order to preserve a producer's last sequence number. One oddity here is that the baseTimestamp field is not preserved during compaction, so it will change if the first record in the batch is compacted away.
The log allows serial appends which always go to the last file. This file is rolled over to a fresh file when it reaches a configurable size (say 1GB). The log takes two configuration parameters: M, which gives the number of messages to write before forcing the OS to flush the file to disk, and S, which gives a number of seconds after which a flush is forced. This gives a durability guarantee of losing at most M messages or S seconds of data in the event of a system crash.
You can run many such mirroring processes to increase throughput and for fault-tolerance (if one process dies, the others will take overs the additional load). Data will be read from topics in the source cluster and written to a topic with the same name in the destination cluster. In fact the mirror maker is little more than a Kafka consumer and producer hooked together. The source and destination clusters are completely independent entities: they can have different numbers of partitions and the offsets will not be the same. For this reason the mirror cluster is not really intended as a fault-tolerance mechanism (as the consumer position will be different); for that we recommend using normal in-cluster replication. The mirror maker process will, however, retain and use the message key for partitioning so order is preserved on a per-key basis. Here is an example showing how to mirror a single topic (named my-topic) from an input cluster: > bin/kafka-mirror-maker.sh --consumer.config consumer.properties --producer.config producer.properties --whitelist my-topic Note that we specify the list of topics with the --whitelist option. This option allows any regular expression using Java-style regular expressions. So you could mirror two topics named A and B using --whitelist 'AB'. Or you could mirror all topics using --whitelist '*'. Make sure to quote any regular expression to ensure the shell doesn't try to expand it as a file path. For convenience we allow the use of ',' instead of '' to specify a list of topics. Sometimes it is easier to say what it is that you don't want. Instead of using --whitelist to say what you want to mirror you can use --blacklist to say what to exclude. This also takes a regular expression argument. However, --blacklist is not supported when the new consumer has been enabled (i.e. when bootstrap.servers has been defined in the consumer configuration). Combining mirroring with the configuration auto.create.topics.enable=true makes it possible to have a replica cluster that will automatically create and replicate all data in a source cluster even as new topics are added. Checking consumer position Sometimes it's useful to see the position of your consumers. We have a tool that will show the position of all consumers in a consumer group as well as how far behind the end of the log they are. To run this tool on a consumer group named my-group consuming a topic named my-topic would look like this: > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-topic 0 2 4 2 consumer-1-029af89c-873c-4751-a720-cefd41a669d6 /127.0.0.1 consumer-1 my-topic 1 2 3 1 consumer-1-029af89c-873c-4751-a720-cefd41a669d6 /127.0.0.1 consumer-1 my-topic 2 2 3 1 consumer-2-42c1abd4-e3b2-425d-a8bb-e1ea49b29bb2 /127.0.0.1 consumer-2 This tool also works with ZooKeeper-based consumers: > bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group my-group Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API). TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID my-topic 0 2 4 2 my-group_consumer-1 my-topic 1 2 3 1 my-group_consumer-1 my-topic 2 2 3 1 my-group_consumer-2 Managing Consumer Groups With the ConsumerGroupCommand tool, we can list, describe, or delete consumer groups. When using the new consumer API (where the broker handles coordination of partition handling and rebalance), the group can be deleted manually, or automatically when the last committed offset for that group expires. Manual deletion works only if the group does not have any active members. For example, to list all consumer groups across all topics: > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list test-consumer-group To view offsets, as mentioned earlier, we "describe" the consumer group like this: > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID topic3 0 241019 395308 154289 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 topic2 1 520678 803288 282610 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 topic3 1 241018 398817 157799 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 topic1 0 854144 855809 1665 consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 topic2 0 460537 803290 342753 consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 topic3 2 243655 398812 155157 consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 There are a number of additional "describe" options that can be used to provide more detailed information about a consumer group that uses the new consumer API: --members: This option provides the list of all active members in the consumer group. > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members CONSUMER-ID HOST CLIENT-ID #PARTITIONS consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 2 consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 1 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 3 consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee /127.0.0.1 consumer3 0
--members --verbose: On top of the information reported by the "--members" options above, this option also provides the partitions assigned to each member. > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1 consumer1 2 topic1(0), topic2(0) consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1 consumer4 1 topic3(2) consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1 consumer2 3 topic2(1), topic3(0,1) consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee /127.0.0.1 consumer3 0 -
--offsets: This is the default describe option and provides the same output as the "--describe" option.
--state: This option provides useful group-level information. > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS localhost:9092 (0) range Stable 4
To manually delete one or multiple consumer groups, the "--delete" option can be used: > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group Note: This will not show information about old Zookeeper-based consumers. Deletion of requested consumer groups ('my-group', 'my-other-group') was successful. If you are using the old high-level consumer and storing the group metadata in ZooKeeper (i.e. offsets.storage=zookeeper), pass --zookeeper instead of bootstrap-server: > bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list Expanding your cluster Adding servers to a Kafka cluster is easy, just assign them a unique broker id and start up Kafka on your new servers. However these new servers will not automatically be assigned any data partitions, so unless partitions are moved to them they won't be doing any work until new topics are created. So usually when you add machines to your cluster you will want to migrate some existing data to these machines. The process of migrating data is manually initiated but fully automated. Under the covers what happens is that Kafka will add the new server as a follower of the partition it is migrating and allow it to fully replicate the existing data in that partition. When the new server has fully replicated the contents of this partition and joined the in-sync replica one of the existing replicas will delete their partition's data. The partition reassignment tool can be used to move partitions across brokers. An ideal partition distribution would ensure even data load and partition sizes across all brokers. The partition reassignment tool does not have the capability to automatically study the data distribution in a Kafka cluster and move partitions around to attain an even load distribution. As such, the admin has to figure out which topics or partitions should be moved around. The partition reassignment tool can run in 3 mutually exclusive modes: --generate: In this mode, given a list of topics and a list of brokers, the tool generates a candidate reassignment to move all partitions of the specified topics to the new brokers. This option merely provides a convenient way to generate a partition reassignment plan given a list of topics and target brokers.
--execute: In this mode, the tool kicks off the reassignment of partitions based on the user provided reassignment plan. (using the --reassignment-json-file option). This can either be a custom reassignment plan hand crafted by the admin or provided by using the --generate option
--verify: In this mode, the tool verifies the status of the reassignment for all partitions listed during the last --execute. The status can be either of successfully completed, failed or in progress
Automatically migrating data to new machines The partition reassignment tool can be used to move some topics off of the current set of brokers to the newly added brokers. This is typically useful while expanding an existing cluster since it is easier to move entire topics to the new set of brokers, than moving one partition at a time. When used to do this, the user should provide a list of topics that should be moved to the new set of brokers and a target list of new brokers. The tool then evenly distributes all partitions for the given list of topics across the new set of brokers. During this move, the replication factor of the topic is kept constant. Effectively the replicas for all partitions for the input list of topics are moved from the old set of brokers to the newly added brokers. For instance, the following example will move all partitions for topics foo1,foo2 to the new set of brokers 5,6. At the end of this move, all partitions for topics foo1 and foo2 will only exist on brokers 5,6. Since the tool accepts the input list of topics as a json file, you first need to identify the topics you want to move and create the json file as follows: > cat topics-to-move.json "topics": ["topic": "foo1", "topic": "foo2"], "version":1 Once the json file is ready, use the partition reassignment tool to generate a candidate assignment: > bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate Current partition replica assignment "version":1, "partitions":["topic":"foo1","partition":2,"replicas":[1,2], "topic":"foo1","partition":0,"replicas":[3,4], "topic":"foo2","partition":2,"replicas":[1,2], "topic":"foo2","partition":0,"replicas":[3,4], "topic":"foo1","partition":1,"replicas":[2,3], "topic":"foo2","partition":1,"replicas":[2,3]] Proposed partition reassignment configuration "version":1, "partitions":["topic":"foo1","partition":2,"replicas":[5,6], "topic":"foo1","partition":0,"replicas":[5,6], "topic":"foo2","partition":2,"replicas":[5,6], "topic":"foo2","partition":0,"replicas":[5,6], "topic":"foo1","partition":1,"replicas":[5,6], "topic":"foo2","partition":1,"replicas":[5,6]] The tool generates a candidate assignment that will move all partitions from topics foo1,foo2 to brokers 5,6. Note, however, that at this point, the partition movement has not started, it merely tells you the current assignment and the proposed new assignment. The current assignment should be saved in case you want to rollback to it. The new assignment should be saved in a json file (e.g. expand-cluster-reassignment.json) to be input to the tool with the --execute option as follows: > bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute Current partition replica assignment "version":1, "partitions":["topic":"foo1","partition":2,"replicas":[1,2], "topic":"foo1","partition":0,"replicas":[3,4], "topic":"foo2","partition":2,"replicas":[1,2], "topic":"foo2","partition":0,"replicas":[3,4], "topic":"foo1","partition":1,"replicas":[2,3], "topic":"foo2","partition":1,"replicas":[2,3]] Save this to use as the --reassignment-json-file option during rollback Successfully started reassignment of partitions "version":1, "partitions":["topic":"foo1","partition":2,"replicas":[5,6], "topic":"foo1","partition":0,"replicas":[5,6], "topic":"foo2","partition":2,"replicas":[5,6], "topic":"foo2","partition":0,"replicas":[5,6], "topic":"foo1","partition":1,"replicas":[5,6], "topic":"foo2","partition":1,"replicas":[5,6]] Finally, the --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json (used with the --execute option) should be used with the --verify option: > bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --verify Status of partition reassignment: Reassignment of partition [foo1,0] completed successfully Reassignment of partition [foo1,1] is in progress Reassignment of partition [foo1,2] is in progress Reassignment of partition [foo2,0] completed successfully Reassignment of partition [foo2,1] completed successfully Reassignment of partition [foo2,2] completed successfully Custom partition assignment and migration The partition reassignment tool can also be used to selectively move replicas of a partition to a specific set of brokers. When used in this manner, it is assumed that the user knows the reassignment plan and does not require the tool to generate a candidate reassignment, effectively skipping the --generate step and moving straight to the --execute step For instance, the following example moves partition 0 of topic foo1 to brokers 5,6 and partition 1 of topic foo2 to brokers 2,3: The first step is to hand craft the custom reassignment plan in a json file: > cat custom-reassignment.json "version":1,"partitions":["topic":"foo1","partition":0,"replicas":[5,6],"topic":"foo2","partition":1,"replicas":[2,3]] Then, use the json file with the --execute option to start the reassignment process: > bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute Current partition replica assignment "version":1, "partitions":["topic":"foo1","partition":0,"replicas":[1,2], "topic":"foo2","partition":1,"replicas":[3,4]] Save this to use as the --reassignment-json-file option during rollback Successfully started reassignment of partitions "version":1, "partitions":["topic":"foo1","partition":0,"replicas":[5,6], "topic":"foo2","partition":1,"replicas":[2,3]] The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same expand-cluster-reassignment.json (used with the --execute option) should be used with the --verify option: > bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --verify Status of partition reassignment: Reassignment of partition [foo1,0] completed successfully Reassignment of partition [foo2,1] completed successfully Decommissioning brokers The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet. As such, the admin has to come up with a reassignment plan to move the replica for all partitions hosted on the broker to be decommissioned, to the rest of the brokers. This can be relatively tedious as the reassignment needs to ensure that all the replicas are not moved from the decommissioned broker to only one other broker. To make this process effortless, we plan to add tooling support for decommissioning brokers in the future. Increasing replication factor Increasing the replication factor of an existing partition is easy. Just specify the extra replicas in the custom reassignment json file and use it with the --execute option to increase the replication factor of the specified partitions. For instance, the following example increases the replication factor of partition 0 of topic foo from 1 to 3. Before increasing the replication factor, the partition's only replica existed on broker 5. As part of increasing the replication factor, we will add more replicas on brokers 6 and 7. The first step is to hand craft the custom reassignment plan in a json file: > cat increase-replication-factor.json "version":1, "partitions":["topic":"foo","partition":0,"replicas":[5,6,7]] Then, use the json file with the --execute option to start the reassignment process: > bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute Current partition replica assignment "version":1, "partitions":["topic":"foo","partition":0,"replicas":[5]] Save this to use as the --reassignment-json-file option during rollback Successfully started reassignment of partitions "version":1, "partitions":["topic":"foo","partition":0,"replicas":[5,6,7]] The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same increase-replication-factor.json (used with the --execute option) should be used with the --verify option: > bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify Status of partition reassignment: Reassignment of partition [foo,0] completed successfully You can also verify the increase in replication factor with the kafka-topics tool: > bin/kafka-topics.sh --zookeeper localhost:2181 --topic foo --describe Topic:fooPartitionCount:1ReplicationFactor:3Configs: Topic: fooPartition: 0Leader: 5Replicas: 5,6,7Isr: 5,6,7 Limiting Bandwidth Usage during Data Migration Kafka lets you apply a throttle to replication traffic, setting an upper bound on the bandwidth used to move replicas from machine to machine. This is useful when rebalancing a cluster, bootstrapping a new broker or adding or removing brokers, as it limits the impact these data-intensive operations will have on users.
2ff7e9595c
Comments